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Preface 


This book is targeted primarily toward engineers and engineering students of ad- 
vanced standing (sophomores, seniors and graduate students). Familiarity with a 
computer language is required; knowledge of basic engineering mechanics is useful, 
but not essential. 

The text attempts to place emphasis on numerical methods, not programming. 
Most engineers are not programmers, but problem solvers. They want to know what 
methods can be applied to a given problem, what are their strengths and pitfalls and 
how to implement them. Engineers are not expected to write computer code for basic 
tasks from scratch; they are more likely to utilize functions and subroutines that have 
been already written and tested. Thus programming by engineers is largely confined 
to assembling existing pieces of code into a coherent package that solves the problem 
at hand. 

The “piece” of code is usually a function that implements a specific task. For the 
user the details of the code are unimportant. What matters is the interface (what goes 
in and what comes out) and an understanding of the method on which the algorithm 
is based. Since no numerical algorithm is infallible, the importance of understanding 
the underlying method cannot be overemphasized; it is, in fact, the rationale behind 
learning numerical methods. 

This book attempts to conform to the views outlined above. Each numerical 
method is explained in detail and its shortcomings are pointed out. The examples that 
follow individual topics fall into two categories: hand computations that illustrate the 
inner workings of the method and smallprograms that show how the computer code is 
utilized in solving a problem. Problems that require programming are marked with ■. 

The material consists of the usual topics covered in an engineering course on 
numerical methods: solution of equations, interpolation and data fitting, numerical 
differentiation and integration, solution of ordinary differential equations and eigen- 
value problems. The choice of methods within each topic is tilted toward relevance 
to engineering problems. For example, there is an extensive discussion of symmetric, 
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Preface 


sparsely populated coefficient matrices in the solution of simultaneous equations. 
In the same vein, the solution of eigenvalue problems concentrates on methods that 
efficiently extract specific eigenvalues from banded matrices. 

An important criterion used in the selection of methods was clarity. Algorithms 
requiring overly complex bookkeeping were rejected regardless of their efficiency and 
robustness. This decision, which was taken with great reluctance, is in keeping with 
the intent to avoid emphasis on programming. 

The selection of algorithms was also influenced by current practice. This disqual- 
ified several well-known historical methods that have been overtaken by more recent 
developments. For example, the secant method for finding roots of equations was 
omitted as having no advantages over Brent’s method. For the same reason, the mul- 
tistep methods used to solve differential equations (e.g., Milne and Adams methods) 
were left out in favor of the adaptive Runge-Kutta and Bulirsch-Stoer methods. 

Notably absent is a chapter on partial differential equations. It was felt that this 
topic is best treated by finite element or boundary element methods, which are outside 
the scope of this book. The finite difference model, which is commonly introduced 
in numerical methods texts, is just too impractical in handling multidimensional 
boundary value problems. 

As usual, the book contains more material than can be covered in a three-credit 
course. The topics that can be skipped without loss of continuity are tagged with an 
asterisk (*). 

The programs listed in this book were tested with Python 2.2.2 and 2.3.4 under 
Windows XP and Red Hat Linux. The source code can be downloaded from the book’s 
website at 


www.cambridge.org/052 1852870 

The author wishes to express his gratitude to the anonymous reviewers and 
Professor Andrew Pytel for their suggestions for improving the manuscript. Credit 
is also due to the authors of Numerical Recipes (Cambridge University Press) whose 
presentation of numerical methods was inspirational in writing this book. 
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General Information 

Quick OverView 

This chapter is not a comprehensive manual of Python. Its sole aim is to provide 
sufficient information to give you a good start if you are unfamiliar with Python. If you 
know another computer language, and presumably you do, it is not difficult to pick 
up the rest as you go. 

Python is an object-oriented language that was developed in late 1980s as a 
scripting language (the name is derived from the British television show Monty 
Python’s Flying Circus). Although Python is not as well known in engineering cir- 
cles as some other languages, it has a considerable following in the programming 
community—in fact, Python is considerably more widespread than Fortran. Python 
may be viewed as an emerging language, since it is stili being developed and re- 
fined. In the current state, it is an excellent language for developing engineering 
applications—it possesses a simple elegance that other programming languages can- 
not match. 

Python programs are not compiled into machine code, but are run by an inter¬ 
preter 1 . The great advantage of an interpreted language is that programs can be tested 
and debugged quickly, allowing the user to concentrate more on the principies be- 
hind the program and less on programming itself. Since there is no need to compile, 
link and execute after each correction, Python programs can be developed in a much 
shorter time than equivalent Fortran or C programs. On the negative side, interpreted 
programs do not produce stand-alone applications. Thus a Python program can be 
run only on computers that have the Python interpreter installed. 


1 The Python interpreter also compiles byte code, which helps to speed up execution somewhat. 
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Python has other advantages over mainstream languages that are important in a 
learning environment: 

• Python is open-source Software, which means that it is /ree; it is included in most 
Linux distributions. 

• Python is available for ali major operating Systems (Linux, Unix, Windows, Mac OS 
etc.). A program written on one system runs without modification on ali Systems. 

• Python is easier to learn and produces more readable code than other languages. 

• Python and its extensions are easy to install. 

Development of Python was clearly influenced by Java and C++, but there is also 
a remarkable similarity to MATLAB® (another interpreted language, very popular 
in scientific computing). Python implements the usual concepts of object-oriented 
languages such as classes, methods, inheritance etc. We will forego these concepts 
and use Python strictly as a procedural language. 

To get an idea of the similarities between MATLAB and Python, let us look at the 
codes written in the two languages for solution of simultaneous equations Ax = b by 
Gauss elimination. Here is the function written in MATLAB: 

function [x,det] = gaussElimin(a,b) 
n = length(b); 
for k = l:n-l 

for i = k+l:n 

if a(i,k) '= 0 

lam = a(i,k)/a(k,k); 

a(i,k+l:n) = a(i,k+l:n) - lam*a(k,k+1:n); 
b(i)= b(i) - lam-b(k); 

end 

end 

end 

det = prod(diag(a)); 
for k = n:-1:1 

b(k) = (b(k) - a(k,k+l:n)*b(k+l:n))/a(k,k); 

end 
x = b; 

The equivalent Python function is: 

from numarray import dot 
def gaussElimin(a,b): 
n = len(b) 




3 


1.1 General Information 


for k in range(0,n-1): 

for i in range(k+1,n): 
if a[i,k] != 0.0: 

lam = a [i,k]/a[k,k] 

a[i,k+l:n] = a[i,k+l:n] - lam*a[k,k+1:n] 
b[i] = b[i] - lam*b[k] 
for k in range(n-l,-1,-1) : 

b[k] = (b[k] - dot(a[k,k+1:n],b[k+1:n]))/a[k,k] 
return b 

The command from numarray import dot instructs the interpreter to load 
the function dot (which computes the dot product of two vectors) from the module 
numarray. The colon (:) operator, known as the slicing operator in Python, works the 
same way it does in MATLAB and Fortran90—it defines a section of an array. 

The statement for k = l :n-l in MATLAB creates a loop that is executed with 
lc= 1,2,..., n— 1. The same loop appears in Python as for k in range(n-l). 

Here the function range(n-l) creates the list [0,1,_ n— 2]; k then loops over 

the elements of the list. The differences in the ranges of k reflect the native off- 
sets used for arrays. In Python all sequences have zero offset, meaning that the in¬ 
dex of the first element of the sequence is always 0. In contrast, the native offset in 
MATLAB is 1. 

Also note that Python has no end statements to terminate blocks of code (loops, 
conditionals, subroutines etc.). The body of a block is defined by its indentation ; hence 
indentation is an integral part of Python syntax. 

Like MATLAB, Python is case sensitive. Thus the names n and N would represent 
different objects. 

Obtaining Python 

Python interpreter can be downloaded from the Python Language Website 
www. python. org. It normally comes with a nice code editor called Idle that allows 
you to run programs directly from the editor. For scientific programming we also 
need the Numarray module which contains various tools for array operations. It is 
obtainable from the Numarray Home Page http://www. stsci. edu/resources/ 
software_hardware/numarray. Both sites also provide documentation for down- 
loading. If you use Linux or Mac OS, it is very likely that Python is already installed on 
your machine (but you must stili download Numarray). 

You should acquire other printed material to supplement the on-line documen¬ 
tation. A commendable teaching guide is Python by Chris Fehly, Peachpit Press, CA 
(2002). As a reference, Python Essential Reference by David M. Beazley, New Riders 
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Publishing (2001) is recommended. By the time you read this, newer editions may be 
available. 


1.2 Core Python 

Variables 

In most computer languages the name of a variable represents a value of a given 
type stored in a fixed memory location. The value may be changed, but not the 
type. This it not so in Python, where variables are typed dynamically. The follow- 
ing interactive session with the Python interpreter illustrates this (> > > is the Python 
prompt): 

»> b = 2 # b is integer type 

»> print b 
2 

»> b = b * 2.0 # Now b is float type 

»> print b 

4.0 


The assignment b = 2 creates an association between the name b and the integer 
value 2. The next statement evaluates the expression b * 2.0 and associates the 
resuit with b; the original association with the integer 2 is destroyed. Now b refers to 
th efloating point value 4.0. 

The pound sign (#) denotes the beginning of a comment —all characters between 
# and the end of the line are ignored by the interpreter. 


Strings 

A string is a sequence of characters enclosed in single or double quotes. Strings are 
concalena/ed wi Lh the plus (+) operator, whereas slicing (:) is used to extract a portion 
of the string. Plere is an example: 

»> stringi = 'Press return to exit’ 

»> string2 = 'the program’ 

»> print stringi + ’ ’ + string2 # Concatenation 

Press return to exit the program 
»> print stringi [0 :12] # Slicing 

Press return 
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Astringis an immutable object—its individual characters cannotbe modifiedwith 
an assignment statement and it has a fixed length. An attempt to violate immutability 
will resuit in TypeError, as shown below. 

»> s = 'Press return to exit’ 

»> s[0] = 'p' 

Traceback (most recent call last): 

File ’’<pyshell#l>’’, line 1, in ? 
s[0] = 'p' 

TypeError: object doesn’t support item assignment 


Tuples 

A tuple is a sequence of arbitrary objects separated by commas and enclosed in paren¬ 
theses. If the tuple contains a single object, the parentheses may be omitted. Tuples 
support the same operations as strings; they are also immutable. Here is an example 
where the tuple rec contains another tuple (6,23,68): 

»> rec = (’Smith',’John’,(6,23,68)) # This is a tuple 

»> lastName,firstName,birthdate = rec # Unpacking the tuple 
»> print firstName 
John 

»> birthYear = birthdate [2] 

»> print birthYear 
68 

»> name = rec[l] + ’ ' + rec[0] 

»> print name 
John Smith 
»> print rec [0:2] 

(’Smith', 'John’) 

Lists 

A list is similar to a tuple, but it is mutable, so that its elements and length can be 
changed. A list is identified by enclosing it in brackets. Here is a sampling of operations 
that can be performed on lists: 

»> a = [1.0, 2.0, 3.0] 

»> a. append(4 • 0) 

»> print a 
[1.0, 2.0, 3.0, 4.0] 


# Create a list 

# Append 4.0 to list 
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»> a.insert(0,0.0) # Insert 0.0 in position 0 

»> print a 

[0.0, 1.0, 2.0, 3.0, 4.0] 

»> print len(a) # Determine length of list 

5 

»> a[2:4] = [1.0, 1.0] # Modify selected elements 
»> print a 

[0.0, 1.0, 1.0, 1.0, 1.0, 4.0] 

If a is a mutable object, such as a list, the assignment statement b = a does not 
resuit in a new object b, but simply creates a new reference to a. Thus any changes 
made to b will be reflected in a. To create an independent copy of a list a, use the 
statement c = a [: ], as illustrated below. 


»> a = [1.0, 2.0, 3.0] 
»> b = a 
»> b[0] = 5.0 
»> print a 


# ’b’ is an alias of ’a’ 

# Change 'b’ 


[5.0, 2.0, 3.0] 
»> c = a[ : ] 

»> c[0] = 1.0 
»> print a 


# The change is reflected in 'a’ 

# ’c’ is an independent copy of ’a’ 

# Change 'c’ 


[5.0, 2.0, 3.0] 


# ’a’ is not affected by the change 


Matrices can represented as nested lists with each row being an element of the 
list. Here is a 3 x 3 matrix a in the form of a list: 


»> a = [[1, 2, 3] , \ 
[4, 5, 6], \ 
[7, 8, 9]] 
»> print a[l] 

[4, 5, 6] 

»> print a[l][2] 

6 


# Print second row (element 1) 


# Print third element of second row 


The backslash (\) is Python’s continuatiori character. Recall that Python sequences 
have zero offset, so that a [ 0 ] represents the first row, a [ l ] the second row, etc. With 
very few exceptions we do not use lists for numerical arrays. It is much more convenient 
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to employ array objects provided by the numarray module, (an extension of Python 
language). Array objects will be discussed later. 

Arithmetic Operators 

Python supports the usual arithmetic operators: 


+ 

Addition 

— 

Subtraction 

* 

Multiplication 

/ 

Division 

** 

Exponentiation 

% 

Modular division 


Some of these operators are also defined for strings and sequences as illustrated 
below. 


»> s = 'Helio ’ 

»> t = 'to you’ 

»> a = [1, 2, 3] 

»> print 3*s 

Helio Helio Helio 

»> print 3*a 

[1, 2, 3, 1, 2, 3, 1, 2, 

»> print a + [4, 5] 

[1, 2, 3, 4, 5] 


# Repetition 

# Repetition 
3] 

# Append elements 


»> print s + t # Concatenation 

Helio to you 

»> print 3 + s # This addition makes no sense 

Traceback (most recent call last): 

File ’’<pyshell#9>’’, line 1, in ? 
print n + s 

TypeError: unsupported operand types for +: ’int’ and 'str' 


Python 2.0 and later versions also have augmented assignment operators, such as 
a += b, that are familiar to the users of C. The augmented operators and the equiv- 
alent arithmetic expressions are shown in the following table. 
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a += b 

a = a + b 

a -= b 

a = a - b 

42 

ii 

* 

ctf 

a = a*b 

a /= b 

a = a/b 

a * * = b 

a = a**b 

45 

II 

ctf 

a = a%b 


Comparison Operators 

The comparison (relational) operators return 1 for true and 0 for false. These operators 
are 


< 

Less than 

> 

Greater than 

< = 

Less than or equal to 

> = 

Greater than or equal to 

== 

Equal to 

! = 

Not equal to 


Numbers of different type (integer, floating point etc.) are converted to a common type 
before the comparison is made. Otherwise, objects of different type are considered to 
be unequal. Here are a few examples: 


»> a = 2 # Integer 

»> b = 1 .99 # Floating point 

»> c = ’ 2 ’ # String 

»> print a > b 
1 

»> print a == c 
0 

»> print (a > b) and (a != c) 

1 


»> print (a > b) or (a == b) 
1 
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Conditionals 

The if construet 


if condition: 
block 

exeeutes a block of statements (which must be indented) if the condition returns true. 
If the condition returns false, the block skipped. The if conditional can be followed 
by any number of elif (short for “else if”) constructs 


elif condition-. 
block 

which work in the same manner. The else clause 


else : 

block 

can be used to define the block of statements which are to be exeeuted if none of 
the if-elif clauses are true. The function sign_of_a below illustrates the use of the 
conditionals. 

def sign_of_a(a): 
if a < 0.0: 

sign = 'negative’ 
elif a > 0.0: 

sign = 'positive' 
else : 

sign = ’zero' 
return sign 

a = 1.5 

print 'a is ’ + sign_of_a(a) 

Running the program results in the output 


a is positive 
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Loops 

The while construet 


while condition-. 
block 

exeeutes a block of (indented) statements if the condition is true. After exeeution of 
the block, the condition is evaluated again. If it is stili true, the block is exeeuted again. 
This process is continued until the condition becomes false. The eis e clause 


else: 

bloclc 


can be used to define the block of statements which are to be exeeuted if condition is 
false, ffere is an example that creates the list [1,1 /2,1/3,...]: 

nMax = 5 
n = 1 
a = [] 

while n < nMax: 

a.appendfl.0/n) 
n = n + 1 
print a 

The output of the program is 

[1.0, 0.5, 0.33333333333333331, 0.25] 

We met the f or statement before in Art. 1.1. This statement requires a target and 
a sequence (usually a list) over which the target loops. The form of the construet is 


# Create empty list 

# Append element to list 


for target in sequence-. 
block 

You may add an else clause which is exeeuted after the for loop has finished. The 
previous program could be written with the for construet as 
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nMax = 5 
a = [ ] 

for n in range(1,nMax): 

a.appendCl.0/n) 
print a 


Here n is the target and the list [1,2. nMax- 1 ], created by calling the r ange 

function, is the sequence. 

Any loop can be terminated by the break statement. If there is an else cause 
associated with the loop, it is not executed. The following program, which searches 
for a name in a list, illustrates the use of break and else in conjunction with a for 
loop: 


list = [’Jack’, ’Jill’, ’Tim’, 'Dave’] 

name = eval(raw_input('Type a name: ’)) # Python input prompt 

for i in range(len(list)): 
if list[i] == name: 

print name,’is number’,i + 1,’on the list’ 
break 

else : 

print name,’is not on the list ’ 

Here are the results of two searches: 

Type a name: ’Tim’ 

Tim is number 3 on the list 

Type a name: ’June' 

June is not on the list 


Type Conversion 

If an arithmetic operation involves numbers of mixed types, the numbers are au- 
tomatically converted to a common type before the operation is carried out. Type 
conversions can also achieved by the following functions: 
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int(a) 

Converts a to integer 

long(a) 

Converts a to long integer 

float(a) 

Converts a to floating point 

complex(a) 

Converts to complex a+Oj 

complex(a,b) 

Converts to complex a + bj 


The above functions also work for converting strings to numbers as long as the 
literal in the string represents a valid number. Conversion from float to an integer is 
carried out by truncation, not by rounding off. Here are a few examples: 

»> a = 5 
»> b = -3.6 
»> d = '4.0' 

»> print a + b 
1.4 

»> print int(b) 

-3 

»> print complex(a,b) 

(5-3.6;j) 

»> print float (d) 

4.0 

»> print int(d) # This fails: d is not Int type 
Traceback (most recent call last): 

File ’’<pyshell#7>’’, line 1, in ? 
print int(d) 

ValueError: invalid literal for int(): 4.0 


Mathematical Functions 

Core Python supports only a few mathematical functions. They are: 


abs(a) 

Absolute value of a 

ma xt sequence) 

Largest element of sequence 

min( sequence') 

Smallest element of sequence 

round(a,n) 

Round a to n decimal places 

cmp(a,b) 

f — 1 if a < b 

Returns j 0 if a = b 
l 1 if a > b 
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The majority of mathematical functions are available in the math module. 

Reading Input 

The intrinsic function for accepting user input is 


raw.input ( prompt ) 


It displays the prompt and then reads a line of input which is converted to a string. To 
convert the string into a numerical value use the function 


eval (string) 


The following program illustrates the use of these functions: 


a = raw_input(’Input a: ’) 

print a, type(a) # Print a and its type 

b = eval(a) 

print b,type(b) # Print b and its type 

The function type (a) returns the type of the object a-, it is a very useful tool in 
debugging. The program was run twice with the following results: 

Input a: 10.0 
10.0 <type ’str’> 

10.0 <type ’float’> 

Input a: 11**2 
11**2 ctype 'str’> 

121 <type ’int’> 

A convenient way to input a number and assign it to the variable a is 


a = eval(raw_input (prompt) ) 
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Printing Output 

Output can be displayed with the print statement: 

print objecti, object2, ... 

which converts objecti, object2 etc. to strings and prints them on the same line, sep- 
arated by spaces. The newline character ’ \n ’ can be uses to force a new line. For 
example, 

»> a = 1234.56789 
»> b = [2, 4, 6, 8] 

»> print a,b 
1234.56789 [2, 4, 6, 8] 

»> print ' a = ’ , a, ’ \nb = ’ , b 

a = 1234.56789 
b = [2, 4, 6, 8] 

The modulo operator (%) can be used to format a tuple. The form of the conversion 
statement is 


’ %formatl %format2 • ■ ■ ’ % tuple 

wher e formati, format2 ■ ■ ■ are the format specifications for each object in the tuple. 
Typically used format specifications are 


wd 

Integer 

w.df 

Floating point notation 

w.de 

Exponential notation 


where w is the width of the field and d is the number of digits after the decimal 
point. The output is right-justified in the specified field and padded with blank spaces 
(there are provisions for changing the justification and padding). Here are a couple of 
examples: 

»> a = 1234.56789 
»> n = 9876 
»> print ’%7.2f’ % a 
1234.57 

»> print ’n = %6d’ % n # Pad with 2 spaces 
n = 9876 
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»> print ’n = %06d’ %n # Pad with 2 zeroes 
n = 009876 

»> print '%12.4e %6d’ % (a,n) 

1.2346e+003 9876 

Error Control 

When an error occurs during execution of a program an exception is raised and the 
program stops. Exceptions can be caught with try and except statements: 


try: 

do something 
except error-. 

do something else 


where error is the name of a built-in Python exception. If the exception error is not 
raised, the try blockis executed; otherwise the execution passes to the except block. 
Ali exceptions can be caught by omitting error from the except statement. 

Here is a statement that raises the exception ZeroDivisionError: 

»> c = 12.0/0.0 

Traceback (most recent call last): 

File ’’<pyshell#0>’’, line 1, in ? 
c = 12.0/0.0 

ZeroDivisionError: float division 
This error can be caught by 
try: 

c = 12.0/0.0 

except ZeroDivisionError: 

print 'Division by zero’ 

1.3 Functions and Modules 

Functions 

The structure of a Python function is 

def func_name(paraml, param2 ,...): 
statements 

return returmvalues 
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where paraml, param2,... are the parameters. A parameter can be any Python ob- 
ject, including a function. Parameters may be given default values, in which case the 
parameter in the function call is optional. If the return statement or rei uni values 
are omitted, the function returns the null object. 

The following example computes the first two derivatives of arctan(x) by finite 
differences: 


from math import arctan 

def finite_d±ff(f,x,h=0.0001): # h has a default value 

df =(f(x+h) - f(x-h))/(2.0*h) 
ddf =(f(x+h) - 2.0*f(x) + f(x-h))/h**2 
return df.ddf 
x = 0.5 

df,ddf = finite_diff(arctan,x) # Uses default value of h 
print 'First derivative =’,df 
print 'Second derivative =’,ddf 

Note that arctan is passed to finite_diff as a parameter. The output from the 
program is 

First derivative = 0.799999999573 
Second derivative = -0.639999991892 

If a mutable object, such as a list, is passed to a function where it is modified, the 
changes will also appear in the calling program. Here is an example: 

def squares(a): 

for i in range(len(a)): 
a[i] = a[i]**2 

a = [1, 2, 3, 4] 
squares(a) 
print a 

The output is 


[1, 4, 9, 16] 
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Modules 

It is sound practice to store useful functions in modules. A module is simply a file 
where the functions reside; the name of the module is the name of the file. A module 
can be loaded into a program by the statement 


from module name import * 


Python comes with a large number of modules containing functions and methods 
for various tasks. Two of the modules are described briefly in the next section. Addi- 
tional modules, including graphics packages, are available for downloading on the 
Web. 


1.4 Mathematics Modules 

math Module 

Most mathematical functions are not built into core Python, but are available by 
loading the math module. There are three ways of accessing the functions in a module. 
The statement 


from math import * 


loads all the function definitions in the math module into the current function or 
module. The use of this method is discouraged because it is not only wasteful, but can 
also lead to conflicts with definitions loaded from other modules. 

You can load selected definitions by 


from math import flincl, func2,... 


as illustrated below. 

»> from math import log,sin 
»> print log(sin(0.5) ) 

- 0.735166686385 
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The third method, which is used by the majority of programmers, is to make the 
module available by 


import math 

The functions in the module can then be accessed by using the module name as a 
prefix: 

»> import math 

»> print math.log(math.sin(0.5)) 

-0.735166686385 

The contents of a module can be printed by calling dir ( module ). Here is how to 
obtain a list of the functions in the math module: 

»> import math 
»> dir(math) 

[’__doc__’, ’__name__’, 'acos', 'asin', 'atan', 

’atan2’, 'ceil', 'cos', 'cosh', 'e', ’exp’, 'fabs', 

'floor', ’fmod’, ’frexp’, ’hypot', ’ldexp’, ’log’, 

'loglO', 'modf', 'pi', 'pow', 'sin', 'sinh', ’sqrt’, 

’tan’, ’tanh’] 

Most of these functions are familiar to programmers. Note that the module in¬ 
cludes two constants: n and e. 

cmath Module 

The cmath module provides many of the functions found in the math module, but 
these accept complex numbers. The functions in the module are: 

[’__doc__’, ’__name__’, 'acos', 'acosh', 'asin', 'asinh', 

'atan', 'atanh', 'cos', 'cosh', 'e', 'exp', 'log', 

'loglO', 'pi', 'sin', 'sinh', 'sqrt', 'tan', 'tanh'] 

Here are examples of complex arithmetic: 

»> from cmath import sin 
»> x = 3.0 -4.5 j 
»> y = 1.2 + 0.8 j 
»> z = 0.8 
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»> print x/y 

(-2.56205313375e-016-3.75j) 

»> print sin(x) 

(6.35239299817+44.5526433649j) 

»> print sin(z) 

(0.7173560909+0j) 

1.5 numarray Module 

General Information 

The numarray module 2 is not a part of the Standard Python release. As pointed out 
before, it must be obtained separately and installed (the installation is very easy). The 
module introduces array objects which are similar to lists, but can be manipulated by 
numerous functions contained in the module. The size of the array is immutable and 
no empty elements are allowed. 

The complete set of functions in numarray is too long to be printed in its entirety. 
The list below is limited to the most commonly used functions. 

['Complex’, 'Complex32', ’Complex64’ , ’Float ’ , 

’Float32’, ’Float64’, 'abs', 'arccos' , 

'arccosh', 'arcsin', 'arcsinh', 'arctan', 

’arctan2’, 'arctanh', 'argmax', 'argmin', 

'cos', 'cosh', ’diagonal’, ’dot’, 'e', ’exp’, 

’floor’, 'identity', ’innerproduct’, ’log’, 

'loglO', ’matrixmultiply’, 'maximum', 'minimum', 

'numarray', 'ones', 'pi', ’product’ 'sin', 'sinh', 

'size', ’sqrt’, 'sum', 'tan', 'tanh', 'trace', 

’transpose’, ’zeros’] 


Creating an Array 

Arrays can be created in several ways. One of them is to use the array function to 
turn a list into an array: 


array (list, type = type specificalion) 


2 Numarray is based on an older Python array module called Numeric. Their interfaces and capa- 
bilities are very similar and they are largely compatible. Although Numeric is stili available, it is no 
longer supported. 
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Here are two examples of creating a 2 x 2 array with floating-point elements: 

»> from numarray import array,Float 
»> a = array([[2.0, -1.0],[-1.0, 3.0]]) 

»> print a 
[[ 2 . - 1 .] 

[-1. 3-]] 

»> b = array([[2, -1],[-1, 3]],type = Float) 

»> print b 
[[ 2 . - 1 .] 

[-1. 3.]] 

Other available functions are 


zeros( (diml , dim2') , type = type_speciflcationl> 


which creates a diml x dim2 array and filis it with zeroes, and 

ones C ( diml , dim2) , type = type_specijication) 

which filis the array with ones. The default type in both cases is Int. 
Finally, there is the function 


arange (/rom, to, incrementi) 


which works just like the range function, but returns an array rather than a list. Here 
are examples of creating arrays: 

»> from numarray import arange,zeros,ones,Float 
»> a = arange (2,10,2) 

»> print a 
[2468] 

»> b = arange (2.0,10.0,2.0) 

»> print b 
[2. 4. 6. 8. ] 

»> z = zeros((4)) 

»> print z 
[0000] 
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»> y = ones( (3,3) , type= Float) 
»> print y 


[[ 

1 . 

1 . 

1 .] 

[ 

1 . 

1 . 

1 .] 

[ 

1 . 

1 . 

1 .]] 


Accessing and Changing Array Elements 

I f a is a rank-2 array, then a [i, j] accesses the element in row i and column j, whereas 
a [1] refers to row i. The elements of an array can be changed by assignment as shown 
below. 

»> from numarray import * 

»> a = zeros((3,3),type=Float) 

»> a[0] = [2.0, 3.1, 1.8] # Change a row 

»> a[l,l] = 5.2 # Change an element 

»> a[2,0:2] = [8.0, -3.3] # Change part of a row 
»> print a 
[[ 2. 3.1 1.8] 

[ 0. 5.2 0. ] 

[ 8. -3.3 0. ]] 

Operations on Arrays 

Arithmetic operators work differendy on arrays than they do on tuples and lists—the 
operation is broadcast to all the elements of the array; that is, the operation is applied 
to each element in the array. Here are examples: 

»> from numarray import array 
»> a = array([0.0, 4.0, 9.0, 16.0]) 

»> print a/16.0 

[ 0. 0.25 0.5625 1. ] 

»> print a - 4.0 
[ -4. 0. 5. 12. ] 

The mathematical functions available in numarray are also broadcast, as illus- 
trated below 

»> from numarray import array,sqrt,sin 
»> a = array([1.0, 4.0, 9.0, 16.0]) 
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»> print sqrt(a) 

[ 1. 2. 3. 4. ] 

»> print sin(a) 

[ 0.84147098 -0.7568025 0.41211849 -0.28790332] 

Functions imported from the math module will work on the individual elements, 
of course, but not on the array itself. Flere is an example: 

»> from numarray import array 

»> from math import sqrt 

»> a = array([1.0, 4.0, 9.0, 16.0]) 

»> print sqrt (a [1]) 

2.0 

»> print sqrt (a) 

Traceback (most recent call last): 


TypeError: Only rank-0 arrays can be cast to floats. 

Array Functions 

There are numerous array functions in numarray that perform matrix operations and 
other useful tasks. Flere are a few examples: 

»> from numarray import * 

»> a = array([[ 4.0, -2.0, 1.0], \ 


[-2.0, 4.0, -2.0], \ 

[ 1.0, -2.0, 3.0]]) 


»> b = array([1.0, 4.0, 2.0]) 
»> print dot(b,b) 

21.0 


# Dot product 


»> print matrixmultiply(a,b) # Matrix multiplication 
[ - 2 . 10 . - 1 .] 


»> print diagonal(a) 
[4. 4. 3.] 

»> print diagonal(a, 1) 
[- 2 . - 2 .] 

»> print trace(a) 

11.0 


# Principal diagonal 


# First subdiagonal 


# Sum of diagonal elements 
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»> 

print 

argmax(b) 

# Index of 

largest element 

1 





»> 

print 

identity(3) 

# Identity 

matrix 

[[1 

0 0] 




[0 

1 0] 




[0 

0 1]] 





Copying Arrays 

We explainedbefore that if ais a mutable object, such as alist, the assignment state- 
ment b = a does not resuit in a new object b, but simply creates a new reference to 
a, called a deep copy. This also applies to arrays. To make an independent copy of an 
array a, use the copy method in the numarray module: 


b = a.copyO 


1.6 Scoping of Variables 

Namespace is a dictionary that contains the names of the variables and their values. 
The namespaces are automatically created and updated as a program runs. There are 
three levels of namespaces in Python: 

• Local namespace, which is created when a function is called. It contains the 
variables passed to the function as arguments and the variables created within 
the function. The namespace is deleted when the function terminates. If a variable 
is created inside a function, its scope is the function’s local namespace. It is not 
visible outside the function. 

• A global namespace is created when a module is loaded. Each module has its own 
namespace. Variables assigned in a global namespace are visible to any function 
within the module. 

• Built-in namespace is created when the interpreter starts. It contains the functions 
that come with the Python interpreter. These functions can be accessed by any 
program unit. 

When a name is encountered during execution of a function, the interpreter 
tries to resolve it by searching the following in the order shown: (1) local namespace, 
(2) global namespace, and (3) built-in namespace. If the name cannot be resolved, 
Python raises a NameError exception. 

Since the variables residing in a global namespace are visible to functions within 
the module, it is not necessary to pass them to the functions as arguments (although 
is good programming practice to do so), as the following program illustrates. 
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def divide(): 
c = a/b 

print ’a/b =’,c 


a = 100.0 
b = 5.0 
divide() 

»> 

a/b =20.0 


Note that the variable c is created inside the function divide and is thus not 
accessible to statements outside the function. Hence an attempt to move the print 
statement out of the function fails: 

def divide(): 
c = a/b 

a = 100.0 
b = 5.0 
divide() 
print 'a/b =’ , c 

»> 

Traceback (most recent call last): 

File ’’C: \Python22\scope .py'’, line 8, in ? 
print c 

NameError: name ’c’ is not defined 


1.7 Writing and Running Programs 

When the Python editor Idle is opened, the user is faced with the prompt >>>, in- 
dicating that the editor is in interactive mode. Any statement typed into the edi¬ 
tor is immediately processed upon pressing the enter key. The interactive mode is a 
good way to learn the language by experimentation and to try out new programming 
ideas. 

Opening a new window places Idle in the batch mode, which allows typing and 
saving of programs. One can also use a text editor to enter program lines, but Idle has 
Python-specific features, such as color coding of keywords and automatic indentation, 
that make work easier. Before a program can be run, it must be saved as a Python file 
with the . py extension, e.g., myprog. py. The program can then be executedby typing 
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python myprog. py; in Windows, double-clicking on the program icon will also work. 
But beware: the program window closes immediately after execution, before you get 
a chance to read the output. To prevent this from happening, conclude the program 
with the line 


raw_input(’press return’) 


Double-clicking the program icon also works in Unix and Linux if the first line 
of the program specifles the path to the Python interpreter (or a shell script that 
provides a link to Python). The path name must be preceded by the symbols #!. On my 
computer the path is /usr/bin/python, so that all my programs start with the line 


#!/usr/b in/python 


On multiuser systems the path is usually /usr/local/bin/python. 

When a module is loaded into a program for the first time with the import state- 

ment, it is compiled into bytecode and written in a file with the extension . pyc. The 
next time the program is run, the interpreter loads the bytecode rather than the origi- 
nal Python file. If in the meantime changes have been made to the module, the module 
is automatically recompiled. A program can also be run from Idle using editi run script 

menu, but automatic recompilation of modules will not take place, unless the existing 
bytecode file is deleted and the program window is closed and reopened. 

Python's error messages can sometimes be confusing, as seen in the following 
example: 

from numarray import array 
a = array([1.0, 2.0, 3.0] 
print a 

raw_input(’press return’) 

The output is 

File ’’C: \Python22\test_module .py’’, line 3 
print a 

SyntaxError: invalid syntax 

What could possibly be wrong with the line print a? The answer is nothing. The 
problem is actually in the preceding line, where the closing parenthesis is missing, 
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making the statement incomplete. Consequently, the interpreter views the third line 
as continuation of the second line, so that it tries to interpret the statement 

a = array([1.0, 2.0, 3.0]print a 

The lesson is this: when faced with a SyntaxError, look at the line preceding the 
alleged offender. It can save a lot of frustration. 

It is a good idea to document your modules by adding a docstring the beginning of 
each module. The docstring, which is enclosed in triple quotes, should explain what 
the module does. Here is an example that documents the module error (we use this 
module in several of our programs): 

## module error 
’ ’ ' err(string). 

Prints 'string’ and terminates program. 

import sys 
def err(string): 
print string 

raw_input(’Press return to exit’) 
sys.exit() 

The docstring of a module can be printed with the statement 
print module_name. __doc__ 

For example, the docstring of error is displayed by 

»> import error 
»> print error . __doc__ 
err(string). 

Prints 'string’ and terminates program. 
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Solve the simultaneous equations Ax = b 


2.1 Introduction 

In this chapter we look at the solution of «linear, algebraic equations in n unknowns. 
It is by far the longest and arguably the most important topic in the book. There is a 
good reason for this—it is almost impossible to carry out numerical analysis of any sort 
without encountering simultaneous equations. Moreover, equation sets arising from 
physical problems are often very large, consuming a lot of computational resources. 
It usually possible to reduce the storage requirements and the run time by exploiting 
special properties of the coefficient matrix, such as sparseness (most elements of a 
sparse matrix are zero). Hence there are many algorithms dedicated to the solution of 
large sets of equations, each one being tailored to a particular form of the coefficient 
matrix (symmetric, banded, sparse etc.). A well-known collection of these routines is 
LAPACK—Linear Algebra PACKage, originally written in Fortran77 3 . 

We cannot possibly discuss all the special algorithms in the limited space avail- 
able. The best we can do is to present the basic methods of solution, supplemented 
by a few useful algorithms for banded and sparse coefficient matrices. 

Notation 

A system of algebraic equations has the form 


AuXi + A 12 X 2 + • • ■ + A\ n x n — b\ 


3 LAPACK is the successor of LINPACK, a 1970s and 80s collection of Fortran subroutines. 
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A 21 X 1 + A 22 X 2 H-+ A 2n x n = b 2 


( 2 . 1 ) 


An\X\ + A n 2 X 2 + • • • + A nn x n — b n 


where the coefficients Ajj and the constants bj are known, and x, represent the un- 
knowns. In matrix notation the equations are written as 


or, simply 


‘An A u ■ 
A 2 i A 22 ■ 

-Ain 
• Azn 


■ & 

_1 


• SS- 

_1 

_A n 1 Ara ■ 

A-nn_ 


J^n_ 


1 


Ax = b 


( 2 . 2 ) 


(2.3) 


A particularly useful representation of the equations for computational purposes 
is the augmented coefficient matrix obtained by adjoining the constant vector b to the 
coefficient matrix A in the following fashion: 



^11 

A\2 • 

■ A\ n 

h 

M- 

A2I 

A 2 2 ’ 

■ A2n 

h 


_An\ 

Ara ■ 

■ A nn 

V 


Uniqueness of Solution 

A system of n linear equations in n unknowns has a unique solution, provided that 
the determinant of the coefficient matrix is nonsingular ; that is, | A / 0. The rows and 
columns of a nonsingular matrix are linearly independent in the sense that no row (or 
column) is a linear combination of other rows (or columns). 

If the coefficient matrix is singular, the equations may have an infinite number of 
Solutions, or no Solutions at ali, depending on the constant vector. As an illustration, 
take the equations 


2x + y = 3 4x + 2y = 6 

Since the second equation can be obtained by multiplying the first equation by two, 
any combination of x and y that satisfies the first equation is also a solution of the 
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second equation. The number of such combinations is infinite. On the other hand, 
the equations 


2x + y= 3 4x + 2y = 0 

have no solution because the second equation, being equivalent to 2x + y = 0, con- 
tradicts the first one. Therefore, any solution that satisfies one equation cannot satisfy 
the other one. 

Ill-Conditioning 

An obvious question is: what happens when the coefficient matrix is almost singular; 
i. e., if ] A| is very small? In order to determine whether the determinant of the coefficient 
matrixis “small,” we need a reference against which the determinant can be measured. 
This reference is called the norm of the matrix and is denoted by || A||. We can then say 
that the determinant is small if 


|A| « ||A|| 

Several norms of a matrix have been defined in existing literature, such as 

n n n 

II A|| = Y^J2 A2 ij IIAII = max^|A y | (2.5a) 

\J /=i J =i -‘- n j= i 

A formal measure of conditioning is the matrix condition number, defined as 

cond(A) = ||A|| |A —1 1 (2.5b) 

If this number is close to unity, the matrix is well-conditioned. The condition number 
increases with the degree of ill-conditioning, reaching infinity for a singular matrix. 
Note that the condition number is not unique, but depends on the choice of the matrix 
norm. Unfortunately, the condition number is expensive to compute for large matri¬ 
ces. In most cases it is sufficient to gauge conditioning by comparing the determinant 
with the magnitudes of the elements in the matrix. 

If the equations are ill-conditioned, small changes in the coefficient matrix resuit 
in large changes in the solution. As an illustration, consider the equations 

2x + y= 3 2x + l.OOly = 0 

that have the solution x = 1501.5, y = —3000. Since |A| = 2(1.001) — 2(1) = 0.002 is 
much smaller than the coefficients, the equations are ill-conditioned. The effect of 
ill-conditioning can be verified by changing the second equation to 2x + 1.002y = 0 
and re-solving the equations. The resuit is x = 751.5, y= -1500. Note that a 0.1% 
change in the coefficient of y produced a 100% change in the solution! 
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Numerical Solutions of ill-conditioned equations are not to be trusted. The reason 
is that the inevitable roundoff errors during the solution process are equivalent to in- 
troducing small changes into the coefficient matrix. This in turn introduces large errors 
into the solution, the magnitude ofwhich depends on the severity of ill-conditioning. 
In suspect cases the determinant of the coefficient matrix should be computed so that 
the degree of ill-conditioning can be estimated. This can be done during or after the 
solution with only a small computational effort. 

Linear Systems 

Linear, algebraic equations occur in almost all branches of numerical analysis. But 
their most visible application in engineering is in the analysis of linear Systems (any 
system whose response is proportional to the input is deemed to be linear). Linear 
Systems include structures, elastic solids, heat flow, seepage of fluids, electromagnetic 
fields and electric circuits, i.e., most topics taught in an engineering curriculum. 

If the system is discrete, such as a truss or an electric Circuit, then its analysis 
leads directly to linear algebraic equations. In the case of a statically determinate 
truss, for example, the equations arise when the equilibrium conditions of the joints 
are written down. The unknowns jq, jc 2 ,..., x„ represent the forces in the members 

and the support reactions, and the constants bi , bz . b n are the prescribed external 

loads. 

The behavior of continuous Systems is described by differential equations, rather 
than algebraic equations. However, because numerical analysis can deal only with 
discrete variables, it is first necessary to approximate a differential equation with a 
system of algebraic equations. The well-known finite difference, finite element and 
boundary element methods of analysis work in this manner. They use different ap- 
proximations to achieve the "discretization,” but in each case the final task is the same: 
solve a system (often a very large system) of linear, algebraic equations. 

In summary, the modeling of linear Systems invariably gives rise to equations of 
the form Ax = b, where b is the input and x represents the response of the system. The 
coefficient matrix A, which reflects the characteristics of the system, is independent 
of the input. In other words, if the input is changed, the equations have to be solved 
again with a different b, but the same A. Therefore, it is desirable to have an equa¬ 
tion solving algorithm that can handle any number of constant vectors with minimal 
computational effort. 

Methods of Solution 

There are two classes of methods for solving Systems of linear, algebraic equations: 
direct and iterative methods. The common characteristic of direct methods is that they 
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transform the original equations into equivalent equations (equations that have the 
same solution) that can be solved more easily. The transformation is carried out by 
applying the three operations listed below. These so-called elementary operations do 
not change the solution, but they may affect the determinant of the coefficient matrix 
as indicated in parenthesis. 

• Exchanging two equations (changes sign of |A|). 

• Multiplying an equation by a nonzero constant (multiplies |A| by the same 
constant). 

• Multiplying an equation by a nonzero constant and then subtracting it from an- 
other equation (leaves ]A| unchanged). 

Iterative, or indirect methods, start with a guess of the solution x, and then re- 
peatedly reline the solution until a certain convergence criterion is reached. Iterative 
methods are generally less efficient than their direct counterparts due to the large 
number of iterations required. But they do have significant computational advan- 
tages if the coefficient matrix is very large and sparsely populated (most coefficients 
are zero). 

OverView of Direct Methods 

Table 2.1 lists three popular direct methods, each of which uses elementary operations 
to produce its own final form of easy-to-solve equations. 


Method 

Initial form 

Final form 

Gauss elimination 

Ax = b 

Ux = c 

LU decomposition 

Ax = b 

LUx = b 

Gauss-Jordan elimination 

Ax = b 

Ix = c 


Table 2.1 


In the above table U represents an upper triangular matrix, L is a lower triangular 
matrix and I denotes the identity matrix. A square matrix is called triangular if it 
contains only zero elements on one side of the leading diagonal. Thus a 3 x 3 upper 
triangular matrix has the form 


Uu 

0 

0 


t/l3 

Un 

U 33 


u = 


Ui2 

U22 

0 
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and a 3 x 3 lower triangular matrix appears as 


L = 


Ln 

L21 

L 31 


0 

L 22 

L 32 


0 

0 

L 33 


Triangular matrices play an important role in linear algebra, since they simplify 
many computations. For example, consider the equations Lx = c, or 


L11X1 = ci 


L21X1 + L22X2 = C2 
L 31 X 1 + L 32 X 2 + L 33 X 3 = C 3 


If we solve the equations forward, starting with the first equation, the computations 
are very easy, since each equation contains only one unknown at a time. The solution 
would thus proceed as follows: 


X\ = Ci/L n 

X 2 = (C 2 — L 2 \X\)/L 22 

X 3 = ( C 3 — L 31 XI — L 32 X 2 ML 33 


T hisprocedureisknovvn as forward substitution. In asimilarway, Ux = c, encountered 
in Gauss elimination, can easily be solved by back substitution, which starts with the 
last equation and proceeds backward through the equations. 

The equations LUx = b, which are associated with LU decomposition, can also 
be solved quickly if we replace them with two sets of equivalent equations: Ly = b 
and Ux = y. Now Ly = b can be solved for y by forward substitution, followed by the 
solution of Ux = y by means of back substitution. 

The equations Ix = c, which are produced by Gauss-Jordan elimination, are 
equivalent to x = c (recall the identity Ix = x), so that c is already the solution. 

EXAMPLE 2.1 

Determine whether the following matrix is singular: 


2.1 

-0.6 

1.1 

3.2 

4.7 

-0.8 

3.1 

-6.5 

4.1 
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Solution Laplace’s development of the determinant (see AppendixA2) about the first 
row of A yields 


|A| = 2.1 


4.7 

-6.5 


- 0.8 

4.1 


(- 0 . 6 ) 


3.2 

3.1 


- 0.8 

4.1 


+ 1.1 


3.2 

3.1 


4.7 

-6.5 


= 2.1(14.07) + 0.6(15.60) + l.l(-35.37) = 0 


Since the determinant is zero, the matrix is singular. It can be verified that the singu- 
larity is due to the following row dependency: (row 3) = (3 x row 1) — (row 2). 

EXAMPLE 2.2 

Solve the equations Ax = b, where 


8 

-6 

2 


28“ 

-4 

11 

-7 

b = 

-40 

4 

-7 

6 
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knowing that the LU decomposition of the coefficient matrix is (you should verify this) 



'2 0 o“ 


“4 -3 1 

A= LU = 

-1 2 0 


0 4-3 


1 -1 1 


-1 

<N 

O 

O 

_1 


Solution We first solve the equations Ly = b by forward substitution: 

2j4 =28 yi = 28/2 = 14 

-yi + 2y 2 = -40 y 2 = (-40 + yO/2 = (-40 + 14)/2 = -13 
H - # + J*s = 33 y^ = 33 — J4 + y 2 = 33 — 14 — 13 = 6 

The solution x is then obtained from Ux = y by back substitution: 

2x 3 = x 3 = y 3 /2 = 6/2 = 3 
4X2 - 3x 3 = y 2 x 2 = (y 2 + 3x 3 )/4 = [-13 + 3(3)] /4 = -1 
4xi - 3x 2 + x 3 = yi Xi = (yi + 3x 2 - x 3 )/4 = [14 + 3(—1) - 3] /4 = 2 

Hence the solution is x = [2 -1 3] 


2.2 Gauss Elimination Method 
Introduction 

Gauss elimination is the most familiar method for solving simultaneous equations. It 
consists of two parts: the elimination phase and the solution phase. As indicated in 
Table 2.1, the function of the elimination phase is to transform the equations into the 
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form Ux = c. The equations are then solved by back substitution. In order to illustrate 
the procedure, let us solve the equations 


4jci — 2x 2 + x 3 = 11 (a) 

-2jci + 4x 2 — 2 x 3 = —16 (b) 

X \ — 2x 2 + 4x 3 = 17 (c) 

Elimination phase The elimination phase utilizes only one of the elementary op- 
erations listed in Table 2.1—multiplying one equation (say, equation j) by a constant 
X and subtracting it from another equation (equation i) . The symbolic representation 
of this operation is 

Eq. (i) <- Eq. (i) — X x Eq. ( j) (2.6) 

The equation being subtracted, namely Eq. (j), is called the pivot equation. 

We start the elimination by taking Eq. (a) to be the pivot equation and choosing 
the multipliers X so as to eliminate x\ from Eqs. (b) and (c): 

Eq. (b) 4- Eq. (b) - (- 0.5) x Eq. (a) 

Eq. (c) <- Eq. (c) - 0.25 x Eq. (a) 

After this transformation, the equations become 


4xi — 2x 2 + x 3 = 11 

(a) 

3x 2 - 1.5x 3 = -10.5 

(b) 

1.5x 2 + 3.75x 3 = 14.25 

(c) 


This completes the first pass. Now we pick (b) as the pivot equation and eliminate x 2 
from (c): 


Eq. (c) Eq. (c) - (- 0.5) x Eq.(b) 
which yields the equations 

4xi - 2x 2 + x 3 = 11 (a) 

3x 2 - 1.5x 3 = -10.5 (b) 

3x 3 = 9 (c) 

The elimination phase is now complete. The original equations have been replaced 
by equivalent equations that can be easily solved by back substitution. 

As pointed out before, the augmented coefficient matrix is a more conve¬ 
nient instrument for performing the computations. Thus the original equations 
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would be written as 


4 

-2 

1 

lf 

-2 

4 

-2 

-16 

1 

-2 

4 

17 


and the equivalent equations produced by the first and the second passes of Gauss 
elimination would appear as 


"4 

-2 

1 

11.00“ 

0 

3 

-1.5 

-10.50 

0 

-1.5 

3.75 

14.25 


“4 

-2 

1 

11.0“ 

0 

3 

-1.5 

-10.5 

0 

0 

3 

9.0 


It is important to note that the elementary row operation in Eq. (2.6) leaves the de¬ 
terminant of the coefficient matrix unchanged. This is rather fortunate, since the de¬ 
terminant of a triangular matrix is very easy to compute—it is the product of the 
diagonal elements. In other words, 

|A| = |U| = U\i X U 22 x • • • x U, m (2.7) 


Back substitution phase The unknowns can now be computed by back substitu- 
tion in the manner described in the previous article. Solving Eqs. (c), (b) and (a) in 
that order, we get 


x 3 = 9/3 = 3 


x 2 = (-10.5+ 1.5 x 3 )/3 = [-10.5+1.5(3)1/3 = -2 
X\ = (11 + 2x 2 - x 3 )/4 = [11 + 2(—2) - 3]/4 = 1 


Algorithm for Gauss Elimination Method 

Elimination phase 

Let us look at the equations at some instant during the elimination phase. Assume that 
the first Arrows of Ahave already been transformed to upper triangular form. Therefore, 
the current pivot equation is the fcth equation, and ali the equations below it are stili to 
be transformed. This situation is depicted by the augmented coefficient matrix shown 
below. Note that the components of A are not the coefficients of the original equations 
(except for the first row), since they have been altered by the elimination procedure. 
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The same applies to the components of the constant vector b. 


^11 

A 12 

A 13 • 

• Aik • 

■ Aij ■ 

■ Ai n 

bi 

0 

A 22 

A 23 ■ 

• Aik ■ 

■ A 2 j ■ 

■ A 2 ,,. 

bi 

0 

0 

A 33 ■ 

■ A3k ■ 

• A 3j ■ 

■ A3„ 

b3 

0 

0 

0 

■ Akk ■ 

■ Akj ■ 

■ Akn 

bk 

0 

0 

0 • 

■ A^ ■ 

■ A] • 

■ A in 

bi 

0 

0 

0 

AfTk 

Anj 

■ A nn 

K 


pivot row 

row being 
transformed 


Let the zth row bea typical row below the pivot equation that is to be transformed, 
meaning that the element is to be eliminated. We can achieve this by multiplying 
the pivot row by k = and subtracting it from the zth row. The corresponding 

changes in the zth row are 


Aij <- Aij — XAkj, j = k, k+ 1 ,..., n (2.8a) 

bi <r- bi - kb k (2.8b) 


To transformthe entire coefficient matrix to uppertriangularform, k and z inEqs. (2.8) 
musthave the ranges k= 1,2 ,... ,n— 1 (chooses the pivotrow), i = k+ 1, k+ 2 ..., n 
(chooses the row to be transformed). The algorithm for the elimination phase now 
almost writes itself: 


for k in range(0,n-1): 

for i in range(k+1,n): 
if a[i,k] != 0.0: 

lam = a[i,k]/a[k,k] 

a[i,k+l:n] = a[i,k+l:n] - lam*a[k,k+1:n] 
b[i] = b[i] - lam*b[k] 

In order to avoid unnecessary operations, the above algorithm departs slightly 
from Eqs. (2.8) in the following ways: 

• If Aik happens to be zero, the transformation of row z is skipped. 

• The index j in Eq. (2.8a) starts with k+ 1 rather than k. Therefore, A^ is not 
replaced by zero, but retains its original value. As the solution phase never accesses 
the lower triangular portion of the coefficient matrix anyway, its contents are 
irrelevant. 
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Back Substitution Phase 

After Gauss elimination the augmented coefficient matrix has the form 


M»] 


An 

Al 2 

A\3 ■ 

Ain 

bC 

0 

A22 

A23 ■ 

• Mn 

bz 

0 

0 

A33 • 

• Asn 

bz 

_ 0 

0 

0 

Afm 



The last equation, A nn x n = b n , is solved first, yielding 

Xn — bn/ A nn 


(2.9) 


Consider now the stage of back substitution where x n , x„_i,..., x > + 1 have been 
already been computed (in that order), and we are about to determine xt from the /cth 
equation 


AkkXk + A^k+lXk+l + • • ' T" AknX n — bk 


The solution is 


Xk = ( bic - ^2 AkjXj k = n- 1, n- 2,..., 1 

\ j=k+l ) Akk 

The corresponding algorithm for back substitution is: 


( 2 . 10 ) 


for k in range(n-l,-1,-1): 

x[k]=(b[k] - dot(a[k,k+l:n],x[k+l:n]))/a[k,k] 


■ gaussElimin 

The function gaussElimin combines the elimination and the back substitution 
phases. During back substitution b is overwritten by the solution vector x, so that 
b contains the solution upon exit. 

## module gaussElimin 
’’' x = gaussElimin(a,b). 

Solves [a]{b> = {x} by Gauss elimination. 

from numarray import dot 

def gaussElimin(a,b): 
n = len(b) 

# Elimination phase 
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for k in range(0,n-1): 

for i in range(k+1,n): 
if a[i,k] != 0.0: 

lam = a [i,k]/a[k,k] 

a[i,k+l:n] = a[i,k+l:n] - lam*a[k,k+1:n] 
b[i] = b[i] - lam*b[k] 

# Back substitution 

for k in range(n-l,-1,-1): 

b[k] = (b[k] - dot(a[k,k+1:n],b[k+1:n]))/a[k,k] 
return b 

Multiple Sets of Equations 

As mentioned before, it is frequently necessary to solve the equations Ax = b for several 
constantvectors. Let therebe msuch constant vectors, denotedbybi, b 2 ,..., b m and 
let the corresponding solution vectors be xi, x 2 ,..., x m . We denote multiple sets of 
equations by AX = B, where 

X = [x x x 2 • • • x m ] B = [bi b 2 • • • b m J 

are n x m matrices whose columns consist of solution vectors and constant vectors, 
respectively. 

An economical way to handle such equations during the elimination phase is 
to include all m constant vectors in the augmented coefficient matrix, so that they 
are transformed simultaneously with the coefficient matrix. The Solutions are then 
obtained by back substitution in the usual manner, one vector at a time. It would 
be quite easy to make the corresponding changes in gaussElimin. However, the LU 
decomposition method, described in the next article, is more versatile in handling 
multiple constant vectors. 

EXAMPLE 2.3 

Use Gauss elimination to solve the equations AX = B, where 


6 

-4 

l" 


"—14 

22" 

-4 

6 

-4 

B = 

36 

-18 

1 

-4 

6 


6 

7 


Solution The augmented coefficient matrix is 


6 

-4 

1 

-14 

22" 

-4 

6 

-4 

36 

-18 

1 

-4 

6 

6 

7 
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The elimination phase consists of the following two passes: 


row2 <- 

row2 + 

(2/3) x 

row 1 

row3 <- 

row3 — 

(1/6) x 

row 1 

‘6 -4 

1 

-14 

22“ 

0 10/3 

-10/3 

80/3 

-10/3 

0 -10/3 

35/6 

25/3 

10/3 

rowl 

<- row 

3 + row 2 

‘6 -4 

1 

-14 

22 

0 10/3 

-10/3 

80/3 

-10/3 

0 0 

5/2 

35 

0 


In the solution phase, we first compute xi by back substitutioni 


35 

Xsi = -= 14 

5/2 

„ 80/3 + (10/3)X 31 80/3 + (10/3)14 oo 

X 2 i = - _ _ - = - , _ - = 22 


10/3 


10/3 


— 14 + 4X21 —X 31 —14 + 4(22) —14 

X11 = ---= ---= tu 


Thus the first solution vector is 


Xl = 


[X„ X21 X 31 ] — 


=10 22 14 


The second solution vector is computed next, also using back substitutioni 


X 32 = 0 


X22 


X 12 


-10/3 + (10/3)X 32 -10/3 + 0 

10/3 ~ HV3 

22 + 4X 22 — X 32 22 + 4(— 1) — 0 

6 ~ 6 


x 2 = 


[x 12 X 22 X 32 ] r =[3 -1 oj 1 


Therefore, 
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EXAMPLE 2.4 


An n x n Vandermode matrix A is defined by 


Aij = v " 1 , i = 1,2 ,..., n, j = l,2,...,n 


where vis avector. Use the function gaussEliminto compute the solution of Ax = b, 
where A is the 6x6 Vandermode matrix generated from the vector 



and 




10 10 


Also evaluate the accuracy of the solution (Vandermode matrices tend to be ill- 
conditioned). 

Solution 

#!/usr/bin/python 
## example2_4 

from numarray import zeros,Float64,array,product, \ 

diagonal,matrixmultiply 
from gaussElimin import * 

def vandermode(v): 
n = len(v) 

a = zeros((n,n),type=Float64) 
for j in range(n): 


a[ : ,j] = v**(n-j-1) 


return a 


v = array([1.0, 1.2, 1.4, 1.6, 1.8, 2.0]) 
b = array([0.0, 1.0, 0.0, 1.0, 0.0, 1.0]) 
a = vandermode(v) 

aOrig = a.copyO # Save original matrix 

bOrig = b.copyO # and the constant vector 

x = gaussElimin(a,b) 

det = product(diagonal(a)) 

print ’x =\n’,x 

print ’\ndet =’,det 
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print ’\nCheck resuit: [a]{x> - b =\n’, \ 


matrixmultiply(aOrig,x) - bOrig 
raw_input(’’ \nPress return to exit’’) 


The program produced the following results: 


X 


[ 416.66666667 -3125.00000004 9250.00000012 -13500.00000017 

9709.33333345 -2751.00000003] 

det = -1.13246207999e-006 

Check resuit: [a]{x> - b = 

[-4.54747351e-13 4.54747351e-13 -1.36424205e-12 4.54747351e-13 

-3.41060513e-ll 9.54969437e-12] 

As the determinant is quite small relative to the elements of A (you may want to 
print A to verify this), we expect detectable roundoff error. Inspection of x leads us to 
suspect that the exact solution is 


9250 -13500 29128/3 -275l] r 



in which case the numerical solution would be accurate to about 10 decimal places. 
Another way to gauge the accuracy of the solution is to compute Ax — b (the resuit 
should be 0). The printout indicates that the solution is indeed accurate to at least 
10 decimal places. 

2.3 LU Decomposition Methods 
Introduction 

It is possible to show that any square matrix A can be expressed as a product of a lower 
triangular matrix L and an upper triangular matrix U: 


A = LU 


( 2 . 11 ) 


The process of computing L and U for a given A is known as LU decomposition or 
LU factorization. LU decomposition is not unique (the combinations of L and U for 
a prescribed A are endless), unless certain constraints are placed on L or U. These 
constraints distinguish one type of decomposition from another. Three commonly 
used decompositions are listed in Table 2.2. 
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Name 

Constraints 

Doolittle’s decomposition 

La =1, i = 1, 2,..., n 

Crout’s decomposition 

Uu = 1, i = 1, 2,..., n 

Choleski’s decomposition 

L = U r 


Table 2.2 


After decomposing A, it is easy to solve the equations Ax = b, as pointed out in 
Art. 2.1. We first rewrite the equations as LUx = b. Upon using the notation Ux = y, 
the equations become 

Ly = b 

which can be solved for y by forward substitution. Then 

Ux = y 

will yield x by the back substitution process. 

The advantage of LU decomposition over the Gauss elimination method is that 
once A is decomposed, we can solve Ax = b for as many constant vectors b as we 
please. The cost of each additional solution is relatively small, since the forward and 
back substitution operations are much less time consuming than the decomposition 
process. 


Doolittle's Decomposition Method 


Decomposition Phase 

Doolittle’s decomposition is closely related to Gauss elimination. In order to illustrate 
the relationship, consider a 3 x 3 matrix A and assume that there exist triangular 
matrices 


" 1 

0 

o" 


‘U11 

U12 

U13" 

^21 

1 

0 

u = 

0 

u 22 

u 23 

_^31 

l 32 

1 


0 

0 

u 33 


such that A = LU. After completing the multiplication on the right hand side, we get 


A = 


U u 

U\\L 2 \ 

U\\L 3 1 


u l2 u 13 

Ui 2 L 2 \ + U 22 U\ 3 L 2 i + U 23 
U\ 2 L 3 i + U 22 L 32 IJ i 3 L 3 1 + u 23 l 32 + u 33 


( 2 . 12 ) 


Let us now apply Gauss elimination to Eq. (2.12). The first pass of the elimina¬ 
tion procedure consists of choosing the first row as the pivot row and applying the 
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elementary operations 


row2 <- row2 — L 2 \ x row 1 (eliminates A 2 1 ) 
row3 <- row 3 — L 3i x row 1 (eliminates A 3 1 ) 


The resuit is 


U u 



U l2 Uia 

U22 U 3 3 

U22 l 32 u 23 l 32 + u 33 


In the next pass we take the second row as the pivot row, and utilize the operation 


row 3 <- row 3 — L 32 x row 2 (eliminates A 32 ) 


ending up with 


A" = U = 


U u 

0 

0 


U\2 U\ 3 
U22 u 23 
0 [/33 


The foregoing illustration reveals two important features of Doolittle’s decompo¬ 
sition: 


• The matrix U is identical to the upper triangular matrix that results from Gauss 
elimination. 

• The off-diagonal elements of L are the pivot equation multipliers used during 
Gauss elimination; that is, Ly is the multiplier that eliminated Ay. 

It is usual practice to store the multipliers in the lower triangular portion of the 
coefficient matrix, replacing the coefficients as they are eliminated (Ly replacing Ay). 
The diagonal elements of L do not have to be stored, since it is understood that each 
of them is unity. The final form of the coefficient matrix would thus be the following 
mixture of L and U: 


[L\U] 


U n 

U l2 

u 13 

L21 

U 22 

u 23 

L31 

L 3 2 

u 33 


(2.13) 


The algorithm for Doolittle’s decomposition is thus identical to the Gauss elimi¬ 
nation procedure in gaussElimin, except that each multiplier X is now stored in the 
lower triangular portion of A: 


for k in range(0,n-1): 

for i in range(k+1,n): 
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if a[i,k] != 0.0: 

lam = a[i,k]/a[k,k] 

a[i,k+l:n] = a[i,k+l:n] - lam*a[k,k+1:n] 
a[i,k] = lam 


Solution Phase 

Consider now the procedure for solving Ly = b by forward substitution. The scalar 
form of the equations is (recall that Lu = i) 

y\ = h 
Lnyi + y 2 = k> 


Lkiy\ + Lia.y2 + • • • + L^k-iyk-i + Tfc — bk 


Solving the A;th equation for yields 

k—1 

yk = bk-J2 Lk jyj' k = 2,3,...,n (2.14) 

j= i 

Therefore, the forward substitution algorithm is 

y[0] = b[0] 

for k in range(l,n): 

y[k] = b[k] - dot(a[k,0:k],y[0:k]) 

The back substitution phase for solving Ux = y is identical to that used in the 
Gauss elimination method. 


■ LUdecomp 

This module contains both the decomposition and solution phases. The decompo- 
sition phase returns the matrix [L\U] shown in Eq. (2.13). In the solution phase, the 
contents of b are replaced by y during forward substitution. Similarly, back substitu¬ 
tion overwrites y with the solution x. 

## module LUdecomp 
’’' a = LUdecomp(a). 

LU decomposition: [L][U] = [a]. The returned matrix 
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[a] = [L\U] contains [U] in the upper triangle and 
the nondiagonal terms of [L] in the lower triangle. 

x = LUsolve(a,b). 

Solves [L][U]{x> = b, where [a] = [L\U] is the matrix 
returned from LUdecomp. 

from numarray import dot 

def LUdecomp(a): 
n = len(a) 

for k in range(0,n-1): 

for i in range(k+1,n): 
if a[i,k] != 0.0: 

lam = a [i,k]/a[k,k] 

a[i,k+l:n] = a[i,k+l:n] - lam*a[k,k+1:n] 
a[i,k] = lam 

return a 

def LUsolve(a,b): 
n = len(a) 

for k in range(l,n): 

b[k] = b[k] - dot(a[k,0:k],b[0:k]) 
for k in range(n-l,-1,-1): 

b[k] = (b[k] - dot(a[k,k+l:n],b[k+l:n]))/a[k,k] 
return b 


Choleski's Decomposition 

Choleski’s decomposition A = LL 7 has two limitations: 

• Since LL 7 is always a symmetric matrix, Choleski’s decomposition requires A to 
be symmetric. 

• The decomposition process involves taking square roots of certain combinations 
of the elements of A. It can be shown that in order to avoid square roots of negative 
numbers A must be positive definite. 

Although the number of multiplications in ali the decomposition methods is 
about the same, Choleski’s decomposition is not a particularly popular means of 
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solving simultaneous equations due to the restrictions listed above. We study it here 
because it is invaluable in certain applications that we encounter later on. 

Let us start by looking at Choleski's decomposition 

A = LL T (2.15) 

of a 3 x 3 matrix: 


An 

^12 

A\3 


^11 

0 

0 " 


Ln 

L 21 

L 31 

^21 

A 22 

A 23 

= 

L21 

L 22 

0 


0 

L 22 

L 32 

^31 

A 32 

^33 


_^31 

L 32 

L 33 


0 

0 

L 33 


After completing the matrix multiplication on the right hand side, we get 


Ali 

A\2 

A3 


r l 2 

Ln L21 

L11L31 

^21 

A22 

A23 

= 

L\i L21 

L 2 + L 2 
^21 ' ^22 

L21 T31 + L22L32 

_A31 

A32 

A33 _ 


cc 

_1 

L21L31 + L22L32 

T 2 + t 2 _ i_ r 2 
^31 ^ ^32 ^ ^33 J 


Note that the right-hand-side matrix is symmetric, as pointed out before. Equating the 
matrices A and LL 7 element-by-element, we obtain six equations (due to symmetry 
only lower or upper triangular elements have to be considered) in the six unknown 
components of L. By solving these equations in a certain order, it is possible to have 
only one unknown in each equation. 

Consider the lower triangular portion of each matrix in Eq. (2.16) (the upper 
triangular portion would do as well). By equating the elements in the first column, 
starting with the first row and proceeding downward, we can compute Ln, L 2 1 and 
Lai in that order: 


An = i-ii Ln = y 7 An 

A 2 i = Ln L 2 i L21 = A 2 i/Ln 

A31 = Ln L31 L31 = A31/L11 

The second column, starting with second row, yields L 2 2 and L 32 : 


A 22 — L 2 1 + L 22 

A32 = L21L31 + L22L32 


L 22 = y A 22 — L 21 
L 32 = (A 32 — L 21 L 31 ) / L 22 


Finally the third column, third row gives us i 33 : 


^33 — ifl + ^32 + ^33 


L 33 — yj A 33 -E33 Z32 
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We can now extrapolate the results for an n x n matrix. We observe that a typical 
element in the lower-triangular portion of LL r is of the form 

J 

(LL ) ij — Lj i Lj i + Lj2 L j2 + • • • + LijLjj — ^ ] Li^Lj^, i > j 

k= i 

Equating this term to the corresponding element of A yields 

j 

Aij =J2 L ikLj k , i = j, 7 + 1. n, 7 = 1,2,...,« (2.17) 

k= i 

The range of indices shown limits the elements to the lower triangular part. For the 
first column (j = 1), we obtain from Eq. (2.17) 


Ln = \! Thi Ln = An/Ln, 1 = 2,3,...,« 


(2.18) 


Proceeding to other columns, we observe that the unknown in Eq. (2.17) is Ly (the 
other elements of L appearing in the equation have already been computed). Taking 
the term containing Ly outside the summation in Eq. (2.17), we obtain 

J -1 

d// = ^ ' LikLjk~\~ 
k=\ 

If i = j (a diagonal term), the solution is 


Ljj — 


\ 


l-i 


Ajj-J2 L %’ J = 2 ’ 3 . 


(2.19D 


fc=l 


For a nondiagonal term we get 


7-i 


fc=i 


L ij =[Aij-^ l L ik Lj k \/Ljj, ; = 2,3.«- 1, i = j + 1, j + 2,..., « (2.20) 


■ choleski(a) 

Before presenting the algorithm for Choleski’s decomposition, we make a useful obser- 
vation: Ay appears only in the formula for Ly. Therefore, once Ly has been computed, 
A[j is no longer needed. This makes it possible to write the elements of L over the lower 
triangular portion of A as they are computed. The elements above the leading diag¬ 
onal of A will remain untouched. The function listed below implements Choleski’s 
decomposition. If a negative diagonal term is encountered during decomposition, an 
error message is printed and the program is terminated. 

## module choleski 
’’’ L = choleski(a) . 
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Choleski decomposition: [L][L]transpose = [a]. 

from numarray import dot 
from math import sqrt 
import error 

def choleski(a): 
n = len(a) 
for k in range(n): 
try: 

a[k,k] = sqrt(a[k,k] - dot(a[k,0:k],a[k,0:k])) 
except ValueError: 

error.err('Matrix is not positive definite’) 
for i in range(k+1,n): 

a[i,k] = (a[i,k] - dot(a[i,0:k],a[k,0:k]))/a[k,k] 
for k in range(l,n): a[0:k,k] = 0.0 
return a 

We could also write the algorithm for forward and back substitutions that are 
necessary in the solution of Ax = b. But since Choleski’s decomposition has no ad- 
vantages over Doolittle’s decomposition in the solution of simultaneous equations, 
we will skip that. 

EXAMPLE 2.5 

Use Doolittle’s decomposition method to solve the equations Ax = b, where 


"l 

4 

l" 


" 7~ 

1 

6 

-1 

b = 

13 

2 

-1 

2 


5 


Solution We first decompose A by Gauss elimination. The first pass consists of the 
elementary operations 

row 2 •<— row 2 — 1 x row 1 (eliminates A 2 1 ) 
row 3 <- row 3 — 2 x row 1 (eliminates A 3 i) 

Storing the multipliers L 2 i = 1 and L 3 1 = 2 in place of the eliminated terms, we obtain 

1 4 f 

1 2 -2 

2-9 0 


A' = 
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The second pass of Gauss elimination uses the operation 

row3 <- row3 — (—4.5) x row2 (eliminates A 32 ) 
Storing the multiplier L 32 = —4.5 in place of A 32 , we get 


A" = [L\U] = 


1 4 1 

1 2 -2 

2 -4.5 -9 


The decomposition is now complete, with 


‘1 

0 

(f 


"1 

4 

l" 

1 

1 

0 

u = 

0 

2 

-2 

2 

-4.5 

1 


0 

0 

-9 


Solution of Ly = b by forward substitution comes next. The augmented coefficient 
form of the equations is 


‘l 

0 

0 

7" 

1 

1 

0 

13 

2 

-4.5 

1 

5 


The solution is 

>1 = 7 

y 2 = 13 - 34 = 13 - 7 = 6 

y 3 = 5 — 2 j 4 + 4.5y 2 = 5 - 2(7) + 4.5(6) = 18 


Finally, the equations Ux = y, or 


"l 

4 

1 

7" 

0 

2 

-2 

6 

0 

0 

-9 

18 


are solved by back substitution. This yields 



*2 


Xl 


6 + 2^3 6 + 2 (— 2 ) ^ 

2 “ 2 _ 

7 — 4x 2 — x 3 = 7 — 4(1) - (-2) = 5 
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EXAMPLE 2.6 

Compute Choleski's decomposition of the matrix 


A = 


4 

-2 

2 


-2 

2 

-4 


2 

-4 

11 


Solution First we note that A is symmetric. Therefore, Choleski’s decomposition is 
applicable, provided that the matrix is also positive definite. An a priori lest for positive 
definiteness is not needed, since the decomposition algorithm contains its own test: if 
the square root of a negative number is encountered, the matrix is not positive definite 
and the decomposition fails. 

Substituting the given matrix for A in Eq. (2.16), we obtain 


4 

-2 

2 


r l 2 

m i 

I 11 I 21 

I 11 I 31 

-2 

2 

-4 

= 

I 11 I 21 

r 2 _r 2 
- l 21 ^ ^22 

I 21 I 31 + I 22 I 32 

2 

-4 

11 


1 

W 

I 21 I 31 + I 22 I 32 

r 2 r 2 _r 2 

^31 ^ ^32 ^ ^33 J 


Equating the elements in the lower (or upper) triangular portions yields 
In = Vi = 2 


L 21 = -2/I n = -2/2 = -1 


Therefore, 


L 3 i = 2/I u =2/2 = 1 
L 2 2 = ^2 - L 2 21 = V2— l 2 = 1 

—4 — I 21 I 3 i —4 — (— 1) (1) 

L 39 = - = - = —O 

l 22 i 

l 33 = yii -^-14 =yn- (D 2 - (-3) 2 =i 


L = 


2 

-1 

1 


0 0 
1 0 
-3 1 


The resuit can easily be verified by performing the multiplication LL 7 . 


EXAMPLE 2.7 

Write a program that solves AX = B with Doolittle’s decomposition method and com¬ 
putes |A|. Utilize the functions LUdecomp and LUsolve. Test the program with 



1 

i-H 

1 

co 

1_ 


1 

A = 

-2 0 5 

B = 

3 2 


1 

ro 

1 

1_ 


1 

1 

cn 

l_ 
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Solution The program listed below decomposes A and then prompts for the constant 
vectors. After a constant vector is entered, the corresponding solution is computed 
and the program prompts for another constant vector. The program is terminated 
when a SyntaxError is encountered in input (e.g., when the “return” key is pressed). 

#!/usr/bin/python 
## example2_7 

from numarray import zeros,array,Float64,product,diagonal 
from LUdecomp import * 

a = array([[ 3.0, -1.0, 4.0], \ 

[-2.0, 0.0, 5.0] , \ 

[ 7.0, 2.0, -2.0]]) 

a = LUdecomp(a) 
det = product(diagonal(a)) 
print ’’ \nDeterminant =’’,det 
while 1: 

print ’’ \nlnput constant vector (press return to exit):’’ 
try: 

b = arrayCeval(raw_input(’’==> ’’)),type=Float64) 
except SyntaxError: break 
x = LUsolve(a,b) 
print ’'The solution is:\n'’,x 
raw_input('’ \nPress return to exit’’) 

Running the program produced the following display: 

Determinant = -77.0 

Input constant vector (press return to exit): 

==> [6.0, 3.0, 7.0] 

The solution is : 

[ 1 . 1 . 1 . ] 

Input constant vector (press return to exit): 

==> [-4.0, 2.0, -5.0] 

The solution is : 

[ -1.00000000e+00 1.00000000e+00 2.30695693e-17] 

Input constant vector (press return to exit): 
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EXAMPLE 2.8 

Test the function choleski by decomposing 


1.44 

-0.36 

5.52 

0.00 

-0.36 

10.33 

-7.78 

0.00 

5.52 

-7.78 

28.40 

9.00 

0.00 

0.00 

9.00 

61.00 


Solution 

#!/usr/bin/python 
## example2_8 

from numarray import array,matrixmultiply,transpose 
from choleski import * 


a = arrayC[[ 

1. 

44 , 

-0. 

36, 

5 . 

. 52, 

0 

.0] , 

\ 

[ 

-0. 

36, 

10. 

33, 

-7. 

. 78, 

0 

.0] , 

\ 

[ 

5 . 

52 , 

-7. 

78, 

28. 

.40, 

9 

.0] , 

\ 

[ 

0. 

0, 

0. 

0, 

9. 

.0, 

61 

.0]] 

) 

L = choleski 

(a) 










print ’L =\n’,L 

print '\nCheck: L*L_transpose =\n', \ 
matrixmultiply(L,transpose(L)) 
raw_input('’ \nPress return to exit’’) 


The output is: 


L = 


[[ 

1 . 2 

0 . 

0 . 

0 . 

] 

[ 

-0.3 

3.2 

0 . 

0 . 

] 

[ 

4.6 

-2 . 

1.8 

0 . 

] 

[ 

0 . 

0 . 

5 . 

6. 

]] 


Check: L*L_transpose = 


[ 

1. 

.44 

-0. 

. 36 

5 . 

52 

0. 

] 

[ 

-0. 

. 36 

10. 

. 33 

-7. 

78 

0. 

] 

[ 

5. 

. 52 

-7. 

. 78 

28 . 

4 

9. 

] 

[ 

0. 


0. 


9 . 


61. 

]] 
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PROBLEM SET 2.1 

1. By evaluating the determinant, classify the following matrices as singular, ill- 
conditioned, or well-conditioned. 


(a) 



2 3 

3 4 

4 5 


2.11 

-0.80 

1.72 

-1.84 

3.03 

1.29 

-1.57 

5.25 

4.30 


(c) 



-1 

2 

-1 


0 

-1 

2 


(d) A = 


3 

-1 

-2 

3 

-18 

13 


2. Given the LU decomposition A = LU, determine A and | A|. 


(a) 



0 0 
1 0 

5/3 1 


U = 


1 2 
0 3 
0 0 


4 

21 

0 


(b) 



0 0 
1 0 

-3 1 


u = 


-1 

1 

0 


1 

-3 

1 


3. Utilize the results of LU decomposition 



" 1 

0 

o" 


"2 

-3 

-1 

A = LU = 

3/2 

1 

0 


0 

13/2 

-7/2 


1/2 

11/13 

1 


0 

0 

32/13 


to solve Ax = b, where b r = ^1 —1 2j. 

4. Use Gauss elimination to solve the equations Ax = b, where 


A = 


2 

3 

2 


-3 -1 
2 -5 
4 -1 


b = 


3 

-9 

-5 


5. Solve the equations AX = B by Gauss elimination, where 


2 

0 

-1 

0' 


"1 

0“ 

0 

1 

2 

0 

B = 

0 

0 

-1 

2 

0 

1 


0 

1 

0 

0 

1 

—2_ 


_0 

0_ 
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6. Solve the equations Ax = b by Gauss elimination, where 


'0 

0 

2 

1 

2' 


' r 

0 

1 

0 

2 

-1 


i 

1 

2 

0 

-2 

0 

b = 

-4 

0 

0 

0 

-1 

1 


-2 

_0 

1 

-1 

1 

-1. 


_-l_ 


Hint: reorder the equations before solving. 

7. Find L and U so that 


A = LU = 


4 

-1 

0 


-1 

4 

-1 


0 

-1 

4 


using (a) Doolittle’s decomposition; (b) Choleski’s decomposition. 

8. Use Doolittle’s decomposition method to solve Ax = b, where 


A = 

-3 

9 

6 

-8 

-4 

24 

b = 

-3 

65 


-12 

24 

-26 


-42 


9. Solve the equations Ax = b by Doolittle’s decomposition method, where 



2.34 

-4.10 

1.78" 


0.02" 

A = 

-1.98 

3.47 

-2.22 

b = 

-0.73 


2.36 

-15.17 

6.18 


-6.63 


10. Solve the equations AX = B by Doolittle’s decomposition method, where 


4 

-3 

6" 


"l 

0“ 

8 

-3 

10 

B = 

0 

1 

-4 

12 

-10 


0 

0 


11. Solve the equations Ax = b by Choleski’s decomposition method, where 


"l 

1 

l' 


" 1 " 

1 

2 

2 

b = 

3/2 

1 

2 

3 


3 


12. Solve the equations 


1 

cn 

1 

CM 

1 

1_ 


Xl 


1.1” 

12 4 -10 


x 2 

= 

0 

-16 28 18 


_x 3 _ 


-2.3 


by Doolittle’s decomposition method. 
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13. Determine L that results from Choleski’s decomposition of the diagonal matrix 


«i 

0 


0 0 

a 2 0 
0 a 3 


14. ■ Modify the function gaussElimin so that it will workwith m constant vectors. 
Test the program by solving AX = B, where 


2 

-1 

o " 


"l 

0 

o ' 

-1 

2 

-1 

B = 

0 

1 

0 

0 

-1 

1 


0 

0 

1 


15. ■ A well-known example of an ill-conditioned matrix is the Hilbert matrix 

” 1 1/2 1/3 •••" 

1/2 1/3 1/4 ••• 

A= 1/3 1/4 1/5 ••• 

Write a program that specializes in solving the equations Ax = b by Doolittle’s 
decomposition method, where A is the Hilbert matrix of arbitrary size nx n, and 

bi = ±A ij 
1=1 

The program should have no input apart from n By running the program, deter¬ 
mine the largest nfor which the solution is within 6 significant figures of the exact 
solution 

-[> > > r 

16. ■ Write a function for the solution phase of Choleskfs decomposition method. 
Test the function by solving the equations Ax = b, where 



4 

-2 

2 


6~ 

A = 

-2 

2 

-4 

b = 

-10 


2 

-4 

11 


27 


Use the function choleski for the decomposition phase. 

17. ■ Determine the coefficients of the polynomial y = a, + a\ x + a 2 x l + a^x 3 that 
pass through the points (0,10), (1, 35), (3, 31) and (4, 2). 

18. ■ Determine the 4th degree polynomial y(x) that passes through the points 
(0, -1), (1, 1), (3, 3), (5, 2) and (6, -2). 
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19. ■ Find the 4th degree polynomial y{x) that passes through the points (0,1), 
(0.75, —0.25) and (1,1), and has zero curvature at (0,1) and (1,1). 

20. ■ Solve the equations Ax = b, where 


3.50 

2.77 

-0.76 

-1.80 

2.68 

3.44 

0.27 

5.07 

6.90 

1.71 

5.45 

2.68 


1.80 


7.31 

0.09 

b = 

4.23 

1.61 


13.85 

1.71 


11.55 


By computing | A| and Ax comment on the accuracy of the solution. 


2.4 Symmetric and Banded Coefficient Matrices 
Introduction 

Engineering problems often lead to coefficient matrices that are sparsely populated, 
meaning that most elements of the matrix are zero. If all the nonzero terms are clus- 
tered about the leading diagonal, then the matrix is said to be banded. An example of 
a banded matrix is 


A = 


'XX 0 0 0" 
X X X 0 0 
0 X X X 0 
0 0 X X X 
_0 0 0 X x_ 


where X’s denote the nonzero elements that form the populated band (some of these 
elements may be zero). All the elements lying outside the band are zero. The matrix 
shown above has a bandwidth of three, since there are at most three nonzero elements 
in each row (or column). Such a matrix is called tridiagonal. 

If a banded matrix is decomposed in the form A = LU, both L and U will retain 
the banded structure of A. For example, if we decomposed the matrix shown above, 
we would get 


'X 

0 

0 

0 

0' 


"X 

X 

0 

0 

0“ 

X 

X 

0 

0 

0 


0 

X 

X 

0 

0 

0 

X 

X 

0 

0 

u = 

0 

0 

X 

X 

0 

0 

0 

X 

X 

0 


0 

0 

0 

X 

X 

_0 

0 

0 

X 

X. 


.0 

0 

0 

0 

x_ 


The banded structure of a coefficient matrix can be exploited to save storage and 
computation time. If the coefficient matrix is also symmetric, further economies are 
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possible. In this section we show how the methods of solution discussed previously 
can be adapted for banded and symmetric coefficient matrices. 


Tridiagonal Coefficient Matrix 

Consider the solution of Ax = b by Doolittle’s decomposition, where A is the n x n 
tridiagonal matrix 


di e\ 0 0 ■ • • 0 

C\ dz> b ■ • • 0 

0 c 2 (k e 3 ■■■ 0 

0 0 C3 d \ ■ • • 0 

0 0 ... 0 Cn -1 d n 


As the notation implies, we are storing the nonzero elements of A in the vectors 


c = 

Cl " 
C 2 

d = 

1 

... ^ 
_1 

e = 

1 

• 

_ 1 


1 

.. i 

0 

_1 


■ 3 * 
_1 


_ @n- 1_ 


The resulting saving of storage can be significant. For example, a 100 x 100 tridiag¬ 
onal matrix, containing 10,000 elements, can be stored in only 99 + 100 + 99 = 298 
locations, which represents a compression ratio of about 33:1. 

Let us now apply LU decomposition to the coefficient matrix. We reduce row k by 
getting rid of Ck-i with the elementary operation 

row k <- row k — (Cfc_i/t4-i) x row (A: — 1), k= 2, 3, ..., n 

The corresponding change in 4t is 

dk <- dic — (Cfc_i/<4-i)efc-i (2.21) 

whereas et is not affected. To finish up with Doolittle’s decomposition of the form 
[L\U], we store the multiplier k = ct-i/ck-i in the location previously occupied by 
Ck— 1 ■ 


Ck— 1 


Ck-l/dk-l 


( 2 . 22 ) 
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Thus the decomposition algorithm is 

for k in range(l,n); 

lam = c[k-l]/d[k-l] 
d[k] = d[k] - lam*e[k-1] 
c[k-l] = lam 

Next we look at the solution phase, i.e., the solution of the Ly = b, followed by 
Ux = y. The equations Ly = b can be portrayed by the augmented coefficient matrix 


1 

0 

0 

0 

0 

h 

Ci 

1 

0 

0 

0 

bi 

0 

c 2 

1 

0 

0 

h 

0 

0 

c 3 

1 ... 

0 

h 

0 

0 


0 c n ~ i 

1 

bn_ 



Note that the original contents of c were destroyed and replaced by the multipliers 
during the decomposition. The solution algorithm for y by forward substitution is 

y[0] = b[0] 

for k in range(l,n): 

y[k] = b[k] - c[k-1]*y[k-l] 

The augmented coefficient matrix representing Ux = y is 


d\ 

ei 

0 • 

0 

0 

yi" 

0 

dz 

e 2 • 

0 

0 

y? 

0 

0 

d% ■ 

0 

0 

ys 

0 

0 

0 • 

dn— i 

@n— 1 

yn-i 

0 

0 

0 • 

0 

d n 

yn 



Note again that the contents of d were altered from the original values during the 
decomposition phase (but e was unchanged). The solution for x is obtained by back 
substitution using the algorithm 

x[n-l] = y[n-l]/d[n-l] 
for k in range(n-2,-1,-1) : 

x[k] = Cy[k] - e[k]*x[k+l])/d[k] 


end do 
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■ LUdecomp3 

This module contains the functions LUdecomp3 and LUsolve3 forthe decomposition 
and solution phases of a tridiagonal matrix. In LUsolve3, the vector y writes over 
the constant vector b during forward substitution. Similarly, the solution vector x 
overwrites y in the back substitution process. In other words, b contains the solution 
upon exit from LUsolve3. 

## module LUdecomp3 

’’’ c,d,e = LUdecomp3(c,d,e). 

LU decomposition of tridiagonal matrix [c\d\e]. On output 
{c},{d} and {e} are the diagonals of the decomposed matrix. 

x = LUsolve3(c,d,e,b). 

Solves [c\d\e]{x> = {b}, where {c}, {d} and {e} are the 
vectors returned from LUdecomp3. 


def LUdecomp3(c,d,e): 
n = len(d) 

for k in range(l,n): 

lam = c[k-l]/d[k-l] 
d[k] = d[k] - lam*e[k-1] 
c[k-l] = lam 
return c,d,e 

def LUsolve3(c,d,e,b) : 
n = len(d) 

for k in range(l,n): 

b[k] = b[k] - c[k-1]*b[k-1] 
b[n-1] = b[n-1]/d[n-1] 
for k in range(n-2,-1,-1): 

b[k] = (b[k] - e[k]*b[k+l])/d[k] 
return b 

Symmetric Coefficient Matrices 

More often than not, coefficient matrices that arise in engineering problems are 
symmetric as well as banded. Therefore, it is worthwhile to discover special prop- 
erties of such matrices and learn how to utilize them in the construction of efficient 
algorithms. 
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If the matrix A is symmetric, then the LU decomposition can be presented in the 
form 


A = LU = LDL r 


(2.23) 


where D is a diagonal matrix. An example is Choleski's decomposition A = LL r that was 
discussed in the previous section (in this case D = I). For Doolittle’s decomposition 
we have 


which gives 



~A o 

0 ••• 

0 ” 


'i 

Ai Ai 

Lnl 


0 A 

0 ••• 

0 


0 

1 A 2 


II 

Ei 

hJ 

Q 

II 

D 

0 0 

A ••• 

0 


0 

0 I 

Lrs 


. 0 0 

0 ••• 

Al 


.0 

0 0 

... 1 _ 



'A 

AAi 

A A 


A A 7 i 




0 

A 

AA 

Af-n2 


U 


0 

0 

A 


A An 




. 0 

0 

0 



Ai . 



(2.24) 


We now see that during decomposition of a symmetric matrix only U has to be stored, 
since D and L can be easily recovered from U. Thus Gauss elimination, which results in 
an upper triangular matrix of the form shown in Eq. (2.24), is sufficient to decompose 
a symmetric matrix. 

There is an alternative storage scheme that can be employed during LU decom¬ 
position. The idea is to arrive at the matrix 



"A 

Ai 

Ai ■ • • 

Ai 


0 

A 

A2 • • • 

A 2 

u* = 

0 

0 

A ••• 

A3 


.0 

0 

0 

D „_ 


Here U can be recovered from A/ = DiLji. It turns out that this scheme leads to a 
computationally more efficient solution phase; therefore, we adopt it for symmetric, 
banded matrices. 
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Symmetric, Pentadiagonal Coefficient Matrix 

We encounter pentadiagonal (bandwidth = 5) coefficient matrices in the solution of 
fourth-order, ordinary differential equations by finite differences. Often these matrices 
are symmetric, in which case an n x «coefficient matrix has the form 



d\ 

ei 

/i 

0 

0 

0 


0 



e \ 

di 

62 

/2 

0 

0 


0 



/i 

e 2 

da, 

63 

/3 

0 


0 


A = 

0 

/2 

63 

d\ 

64 

/4 


0 

(2.26) 


0 


0 

fi- 4 

6/z—3 

dn-2 

e n - 2 

fn-2 



0 


0 

0 

fn- 3 

6«—2 

dn— 1 

&n— 1 



0 


0 

0 

0 

fn- 2 

&n— 1 

d n 


As in the case of tridiagonal matrices, we store 

the nonzero elements in the three 


vectors 


d\ 





dz 


‘ ei ' 

62 


" /l ' 

/2 

dn- 2 
^2— 1 

e = 

6)z-2 

f = 

1 

CN 

... I 

_ 1 

^77 


_6/)-l_ 




Let us now look at the solution of the equations Ax = b by Doolittle’s decomposi- 
tion. The first step is to transform A to upper triangular form by Gauss elimination. If 
elimination has progressed to the stage where the fcth row has become the pivot row, 
we have the following situation: 





... 0 

dk e k fk 

0 0 0 

... 0 

6fc r4+i 6fc + i 

fk+ 1 0 0 ... 

... 0 

fk 6fc+1 r4+2 

6fc+2 fic+2 0 

... 0 

0 il-+l 6 fc + 2 

4:+3 6fc+3 fk+ 3 ■ • • 



. . . • _ 



The elements e;, : and fk below the pivot row (the /ah row) are eliminated by the 
operations 


row {k+ 1) <— row (A:+ 1) — (efc/4t) x row k 
row (fc+ 2) row (fc+ 2) — [fk/dk) x row k 
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The only terms (other than those being eliminated) that are changed by the above 
operations are 


dk +1 <- dk+i - ta/ df) ek 

Sk +1 <- eic+i ~ ( Bk/dk)fk (2.27a) 

dk+2 <- dk+2 - ifk/dk) fk 

Storage of the multipliers in the upper triangular portion of the matrix results in 

ek <r- e k /dk f k <r- fk/dk (2.27b) 

At the conclusion of the elimination phase the matrix has the form (do not confuse d, 


e and f with the original contents of A) 





d\ 

ei 

/l 

0 

0 


0 

d 2 

e 2 

/2 

0 

U* = 

0 

0 


e 3 ... 

0 


0 

0 


0 dfi— i 

@n— 1 


0 

0 


0 0 

d n 


Next comes the solution phase. The equations Ly = b have the augmented coef- 
hcient matrix 


i 

0 

0 

0 

... 0 

h 

e\ 

1 

0 

0 

... 0 

h 

/i 

ez 

1 

0 

... 0 

h 

0 

/2 

e 3 

1 

... 0 

h 

0 

0 

0 

fn—2 

@n— 1 1 

bfi_ 



Solution by forward substitution yields 

yi = h 

y 2 = h - eiyi (2.28) 

yk = h- fic—2yk—2 - ek-iyk-1, k = 3,4, ..., n 
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The equations to be solved by back substitution, namelyUx = y, have the augmented 
coefficient matrix 


d\ 

d\ 6\ 

di fi 

0 

0 

Yi " 

0 

di 

d>e 2 

<kf 2 ••• 

0 

Y2 

0 

0 

da, 

da,e 3 

0 

Y3 

0 

0 


0 d n -\ 

dn— 1 @n— 1 

Yn-1 

0 

0 


0 0 

d/i 

Yn _ 



the solution of which is obtained by back substitution: 

Xn — Yn/ dn 

Xn— 1 — Yn— I / d n - | G n —\X n 

Xk = Yk/dk ~ e k x k+1 - fkX k+ 2 , k= n-2, n-3 ,..., 1 (2.29) 


■ LUdecomp5 

The function LUdecompS below decomposes a symmetric, pentadiagonal matrixA of 
the form A = [f\e\d\e\f]. The original vectors d, e and f are destroyed and replaced 
by the vectors of the decomposed matrix. After decomposition, the solution of Ax = b 
can be obtained by LUs olve 5. During forward substitution, the original b is replaced 
by y. Similarly, y is written over by x in the back substitution phase, so that b contains 
the solution vector upon exit from LUsolveS. 

## module LUdecomp5 

’’’ d,e,f = LUdecomp5(d,e,f). 

LU decomposition of symetric pentadiagonal matrix 
[f\e\d\e\f]. On output {d},{e} and {f} are the 
diagonals of the decomposed matrix. 

x = LUsolve5(d,e,f,b). 

Solves [f\e\d\e\f]{x> = {b} , where {d}, {e} and {f} 
are the vectors returned from LUdecomp5. 

def LUdecomp5(d,e,f): 
n = len(d) 

for k in range(n-2): 
lam = e[k]/d[k] 
d[k+l] = d[k+l] - lam*e[k] 
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e[k+l] = e[k+l] - lam*f[k] 
e[k] = lam 
lam = f[k]/d[k] 
d[k+2] = d[k+2] - lam*f[k] 
f[k] = lam 
lam = e[n-2]/d[n-2] 
d[n-l] = d[n-l] - lam*e[n-2] 
e[n-2] = lam 
return d,e,f 

def LUsolve5(d,e,f,b): 
n = len(d) 

b [ 1] = b [ 1] - e [0] *b [0] 
for k in range(2,n): 

b[k] = b[k] - e[k-1]*b[k-1] - f[k-2]*b[k-2] 
b[n-1] = b[n-1]/d[n-l] 

b[n-2] = b[n-2]/d[n-2] - e[n-2]*b[n-1] 
for k in range(n-3,-1,-1) : 

b[k] = b[k]/d[k] - e[k]*b[k+l] - f[k]-b[k+2] 
return b 


EXAMPLE 2.9 

As a resuit of Gauss elimination, a symmetric matrix A was transformed to the upper 
triangular form 


4-2 1 0 

_ 0 3 -3/2 1 

“ 0 0 3 -3/2 

00 0 35/12 

Determine the original matrix A. 


Solution First we find L in the decomposition A = LU. Dividing each row of U by its 
diagonal element yields 








65 


2.4 Symmetric and Banded Coefficient Matrices 


Therefore, A = LU becomes 



1 

0 

0 

0' 


'4 

-2 

1 

0“ 

A = 

-1/2 

1 

0 

0 


0 

3 

-3/2 

1 


1/4 

-1/2 

1 

0 


0 

0 

3 

-3/2 


0 

1/3 

-1/2 

1_ 


_0 

0 

0 

35/12_ 


4-210 
-2 4-2 1 

1-2 4-2 

0 1-24 


EXAMPLE 2.10 

Determine L and D that resuit from Doolittle’s decomposition A = LDL r of the sym¬ 
metric matrix 


A = 


3 

-3 

3 


-3 

5 

1 


3 

1 

10 


Solution We use Gauss elimination, storing the multipliers in the upper triangular 
portion of A. At the completion of elimination, the matrix will have the form of U* in 
Eq. (2.25). 

The terms to be eliminated in the first pass are A 2 1 and A 33 using the elementary 
operations 


row 2 <- row 2 — (— 1) x row 1 
row3 <- row 3 — (1) x row 1 


Storing the multipliers (— 1 and 1) in the locations occupied by A 12 and Ai 3 , we get 


A' = 


3 

0 

0 


-1 1 
2 4 
4 7 


The second pass is the operation 


row 3 <r- row 3 — 2 x row 2 


which yields, after overwriting A 23 with the multiplier 2 

3 -1 1 
0 2 2 
0 0-1 


A" = [0\D\L r ] 
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Hence 


L = 

EXAMPLE 2.11 


1 

0 

0 


3 

0 

0 

1 

1 

0 

D = 

0 

2 

0 

1 

2 

1 


0 

0 

-1 


Utilize the functions LUdecmp3 and LUsolve3 to solve Ax = b, where 


' 2 

-1 

0 

0 

0' 


' 5" 

-1 

2 

-1 

0 

0 


-5 

0 

-1 

2 

-1 

0 

b = 

4 

0 

0 

-1 

2 

-1 


-5 

0 

0 

0 

-1 

2. 


5_ 


Solution 

#!/usr/bin/python 
## example2_ll 

from numarray import array,ones 
from LUdecomp3 import * 


d = ones((5))*2.0 
c = ones((4))*(-1.0) 

b = array([5.0, -5.0, 4.0, -5.0, 5.0]) 
e = c.copyO 

c,d,e = LUdecomp3(c,d,e) 
x = LUsolve3(c,d,e,b) 
print ’’\nx =\n’’,x 

raw_input(’’ \nPress return to exit’’) 


The output is: 


X = 

[ 2 .- 1 . 1 . - 1 . 2 . ] 

2.5 Pivoting 

Introduction 

Sometimes the order in which the equations are presented to the solution algorithm 
has a profound effect on the results. For example, consider the equations 

2xi — x 2 = 1 
— X\ + 2x 2 — x 3 = 0 
-x 2 + x 3 = 0 
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The corresponding augmented coefficient matrix is 


2 

-1 

0 

1 

1 

t— 1 

2 

-1 

0 

1 

O 

-1 

1 

1 

O 


(a) 


Equations (a) are in the "right order” in the sense that we would have no trouble obtain- 
ing the correct solution Xi = x 2 = x 3 = 1 by Gauss elimination or LU decomposition. 
Now suppose that we exchange the first and third equations, so that the augmented 
coefficient matrix becomes 


1 

O 

-1 

1 

0 

1 

t — 1 

2 

-1 

0 

2 

-1 

0 

- 1 

1 —1 


(b) 


Since we did not change the equations (only their order was altered), the solution is stili 
Xi = x 2 = X 3 = 1. However, Gauss elimination fails immediately due to the presence 
of the zero pivot element (the element /In). 

The above example demonstrates that it is sometimes essential to reorder the 
equations during the elimination phase. The reordering, or row pivoting, is also re- 
quired if the pivot element is not zero, but very small in comparison to other elements 
in the pivot row, as demonstrated by the following set of equations: 


e 

-1 

1 

0' 

-1 

2 

-1 

0 

2 

-1 

0 

1 


(c) 


These equations are the same as Eqs. (b), except that the small number e replaces the 
zero element A n in Eq. (b). Therefore, if we let e 0, the Solutions of Eqs. (b) and (c) 
should become identical. After the first phase of Gauss elimination, the augmented 
coefficient matrix becomes 


e 

-1 

1 

0“ 

0 

2- 1/e 

-1 + 1/e 

0 

0 

-1 + 2/e 

-2/e 

1 


Because the computer works with a fixed word length, all numbers are rounded off 
to a finite number of significant figures. If e is very small, then 1 /e is huge, and an 
element such as 2 — 1/e is rounded to — 1/e. Therefore, for sufficiently small e, the 
Eqs. (d) are actually stored as 


e 

-1 

1 

0“ 

0 

— 1/e 

1/e 

0 

0 

2/e 

-2/e 

1 
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Because the second and third equations obviously contradict each other, the solution 
process fails again. This problem would not arise if the first and second, or the first 
and the third equations were interchanged in Eqs. (c) before the elimination. 

The last example illustrates the extreme case where e was so small that roundoff 
errors resulted in total failure of the solution. If we were to make e somewhat bigger 
so that the solution would not “bomb” any more, the roundoff errors might stili be 
large enough to render the solution unreliable. Again, this difficulty could be avoided 
by pivoting. 

Diagonal Dominance 

An /ix «matrix A is said to be diagonally dominant if each diagonal element is larger 
than the sum of the other elements in the same row (we are talking here about absolute 
values). Thus diagonal dominance requires that 

n 

\Au\ > J2\ A ij\ 0'=1.2.n) (2.30) 

j =i 

For example, the matrix 

—2 4 -l" 

1 -1 3 

4 -2 1 

is not diagonally dominant, but if we rearrange the rows in the following manner 

"4-2 l" 

-2 4 -1 

1 _1 3 _ 

then we have diagonal dominance. 

It can be shown that if the coefficient matrix of the equations Ax = b is diagonally 
dominant, then the solution does not benefit from pivoting; that is, the equations are 
already arranged in the optimal order. It follows that the strategy of pivoting should be 
to reorder the equations so that the coefficient matrixis as close to diagonal dominance 
as possible. This is the principle behind scaled row pivoting, discussed next. 

Gauss Elimination with Scaled Row Pivoting 

Consider the solution of Ax = b by Gauss elimination with row pivoting. Recall that 
pivoting aims at improving diagonal dominance of the coefficient matrix, i.e., making 
the pivot element as large as possible in comparison to other elements in the pivot 
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row. The comparison is made easier if we establish an array s with the elements 

Si = max \Aij \, i =1,2, ...,n (2.31) 

Thus Si, called the scale factor of row i, contains the absolute value of the largest 
element in the /1 h row of A. The vector s can be obtained with the algorithm 

for i in range(n): 

s[i] = max(abs(a[i, : ])) 


The relative size of an element A,j (that is, relative to the largest element in the 
zth row) is defined as the ratio 



Suppose that the elimination phase has reached the stage where the k Lh row has 
become the pivot row. The augmented coefficient matrix at this point is shown below. 


An 

Al2 

A13 

Au ■ ■ ■ 

Aln 

htl 

0 

A22 

A23 

A24 • • • 

A2 n 

bz 

0 

0 

A33 

A34 ■ • • 

A3 „ 

h 

0 


0 

Afcfc • • • 

Afcn 

h 

0 


0 

A,ik ■ ■ • 

A nn 

bn_ 


We don’t automatically accept A** as the next pivot element, but look in the k th column 
below Ajck for a “better” pivot. The best choice is the element A p k that has the largest 
relative size; that is, we choose psuch that 


t P k = max tjk, j >k 

If we find such an element, then we interchange the rows k and p, and proceed with 
the elimination pass as usual. Note that the corresponding row interchange must also 
be carried out in the scale factor array s. The algorithm that does ali this is 


for k in range(0,n-1): 


# Find row containing element with largest relative size 
p = int(argmax(abs(a[k:n,k])/s[k:n])) + k 


# If this element is very small, matrix is singular 
if abs(a[p,k]) < tol: error.err('Matrix is singular’) 
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# Check whether rows k and p must be interchanged 
if p != k: 

# Interchange rows if needed 
swap.swapRows(b,k,p) 
swap.swapRows(s,k,p) 
swap.swapRows(a,k,p) 

# Proceed with elimination 


The Python statement int (argmax (v)) returns the index of the largest element 
in the vector v. The algorithms for exchanging rows (and columns) are included in the 
module swap shown below. 


■ swap 

The function swapRows interchanges rows i and j of a matrix or vector v, whereas 
swapCols interchanges columns i and j of a matrix. 


## module swap 
’ ’ ’ swapRows(v,i,j) . 

Swaps rows i and j of vector or matrix [v]. 
swapCols(v,i,j). 

Swaps columns i and j of matrix [v]. 

def swapRows(v,i,j): 

if len(v.getshape()) == 1: 

v[i],v[j] = v[j],v[i] 
else : 

temp = v[i].copy() 
v[i] = v[j] 
v[ j] = temp 

def swapCols(v,i,j) : 

temp = v[ : , j ] . copyO 
v[ : , j] = v[: ,i] 
v[ : ,i] = temp 





71 


2.5 Pivoting 


■ gaussPivot 

The function gaussPivot performs Gauss elimination with row pivoting. Apart from 
row swapping, the elimination and solution phases are identical to gaussElimin in 
Art. 2.2. 

## module gaussPivot 

’’’ x = gaussPivot(a,b,tol=l.Oe-9). 

Solves [a]{x> = {b} by Gauss elimination with 
scaled row pivoting 

from numarray import * 
import swap 
import error 

def gaussPivot(a,b,tol=l.Oe-9) : 
n = len(b) 

# Set up scale factors 

s = zeros((n),type=Float64) 
for i in range(n): 

s[i] = max(abs(a[i, : ])) 

for k in range(0,n-1): 

# Row interchange, if needed 

p = int(argmax(abs(a[k:n,k])/s[k:n])) + k 
if abs(a[p,k]) < tol: 

error.err(’Matrix is singular’) 
if p != k: 

swap.swapRows(b,k,p) 
swap.swapRows(s,k,p) 
swap.swapRows(a,k,p) 

# Elimination 

for i in range(k+1,n): 
if a[i,k] != 0.0: 

lam = a[i,k]/a[k,k] 

a[i,k+l:n] = a [i,k+l:n] - lam*a[k,k+1:n] 
b[i] = b[i] - lam*b[k] 
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if abs(a[n-1,n-1]) < tol: 

error.err('Matrix is singular’) 

# Back substitution 

for k in range(n-l,-1,-1): 

b[k] = (b[k] - dot(a[k,k+l:n],b[k+l:n]))/a[k,k] 
return b 

■ LUpivot 

The Gauss elimination algorithm can be changed to Doolittle’s decomposition with 
minor changes. The most important of these is keeping a record of the row inter- 
changes during the decomposition phase. In LUdecomp this record is kept in the array 
seq. Initially seq contains [0,1,2,...]. Whenever two rows are interchanged, the cor- 
responding interchange is also carried out in seq. Thus seq shows the order in which 
of the original rows have been rearranged. This information is passed on to the so- 
lution phase (LUsolve), which rearranges the elements of the constant vector in the 
same order before proceeding to forward and back substitutions. 

## module LUpivot 

’’’ a,seq = LUdecompCa,tol=l.Oe-9). 

LU decomposition of matrix [a] using scaled row pivoting. 

The returned matrix [a] = [L\U] contains [U] in the upper 
triangle and the nondiagonal terms of [L] in the lower triangle. 

Note that [L][U] is a row-wise permutation of the original [a]; 
the permutations are recorded in the vector {seq}. 

x = LUsolve(a,b,seq). 

Solves [L][U]{x> = {b} , where the matrix [a] = [L\U] and the 
permutation vector {seq} are returned from LUdecomp. 

from numarray import argmax,abs,dot,zeros,Float64,array 
import swap 
import error 

def LUdecompCa,tol=l.Oe-9): 
n = len(a) 

seq = array(range(n)) 
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# Set up scale factors 

s = zeros((n),type=Float64) 
for i in range(n): 

s[i] = max(abs(a[i,:])) 

for k in range(0,n-1): 

# Row interchange, if needed 

p = int(argmax(abs(a[k:n,k])/s[k:n])) + k 
if abs(a[p,k]) < tol: 

error.err('Matrix is singular’) 
if p != k: 

swap.swapRows(s,k,p) 
swap.swapRows(a,k,p) 
swap.swapRows(seq,k,p) 

# Elimination 

for i in range(k+l,n): 
if a[i,k] != 0.0: 

lam = a[i,k]/a[k,k] 

a[i,k+l:n] = a[i,k+l:n] - lam*a[k,k+1:n] 
a[i,k] = lam 

return a,seq 

def LUsolve(a,b,seq): 
n = len(a) 

# Rearrange constant vector; store it in [x] 
x = b.copyO 

for i in range(n): 
x[i] = b[seq[i]] 

# Solution 

for k in range(l,n): 

x[k] = x[k] - dot(a[k,0:k],x[0:k]) 
for k in range(n-l,-1,-1): 

x[k] = Cx[k] - dot(a[k,k+l:n],x[k+l:n]))/a[k,k] 


return x 
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When to Pivot 

Pivoting has a couple of drawbacks. One of these is the increased time of computation; 
the other is the destruction of the symmetry and banded structure of the coefficient 
matrix. The latter is of particular concern in engineering computing, where the co¬ 
efficient matrices are frequently banded and symmetric, a property that is utilized 
in the solution, as seen in the previous chapter. Fortunately, these matrices are often 
diagonally dominant as well, so that they would not benefit from pivoting anyway. 

There are no infallible rules for determining when pivoting should be used. Expe- 
rience indicates that pivoting is likely to be counterproductive if the coefficient matrix 
is banded. Positive definite and, to a iesser degree, symmetric matrices aiso seidom 
gain from pivoting. And we shouid not forget that pivoting is not the oniy means of 
controliing roundoff errors—there is also double precision arithmetic. 

It should be strongly emphasized that the above rules of thumb are only meant 
for equations that stem from real engineering problems. It is not difficult to concoct 
“textbook” examples that do not conform to these rules. 

EXAMPLE 2.12 

Employ Gauss elimination with scaled row pivoting to solve the equations Ax = b, 
where 


2 

-2 

6' 


' 16“ 

-2 

4 

3 

b = 

0 

-1 

8 

4 


-1 


Solution The augmented coefficient matrix and the scale factor array are 


'2-26 

16' 


"6“ 

-2 4 3 

0 

S = 

4 

-1 8 4 

-1 


8 


Note that s contains the absolute value of the largest element in each row of A. At this 
stage, all the elements in the first column of A are potential pivots. To determine the 
best pivot element, we calculate the relative sizes of the elements in the first column: 


rn 


l^flll /-Sl 


"1/3" 

r 2 1 

= 

1 ^211 /S 2 

= 

1/2 

J31_ 


_l^3ll /$3_ 


1/8 


Since r 2 i is the largest element, we conclude that A 21 makes the best pivot element. 
Therefore, we exchange rows 1 and 2 of the augmented coefficient matrix and the 
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scale factor array, obtaining 


—2 

4 

3 

o~ 


V 

2 

-2 

6 

16 

s = 

6 

-1 

8 

4 

-1 


8 


Now the first pass of Gauss elimination is carried out (the arrow points to the pivot 
row), yielding 


—2 

4 

3 

o‘ 


~4~ 

0 

2 

9 

16 

S = 

6 

0 

6 

5/2 

-1 


8 


The potential pivot elements for the next elimination pass are A 22 and A , a . We 
determine the “winner” from 


* 


* 


* 

t *22 

= 

1 ^22 1 / S 2 

= 

1/3 

>32 


1 A 32 1 / S 3 


_3/4_ 


Note that r\ 2 is irrelevant, since row 1 already acted as the pivot row. Therefore, it is 
excluded from further consideration. As r 32 is larger than r 22 , the third row is the better 
pivot row. After interchanging rows 2 and 3, we have 


[ A ' | b'] = 


"-2 4 3 

0" 


~4~ 

0 6 5/2 

-1 

S = 

8 

0 2 9 

16 


6 


The second elimination pass now yields 


[A" | b"] = [u | c] 




‘-2 4 3 

0 

0 6 5/2 

-1 

0 0 49/6 

49/3 


This completes the elimination phase. It should be noted that U is the matrix that 
would resuit from LU decomposition of the following row-wise permutation of A (the 
ordering of rows is the same as achieved by pivoting): 


-2 4 3 

-1 8 4 

2-2 6 


Since the solution of Ux = c by back substitution is not affected by pivoting, we skip 
the detailed of the computation. The resuit is x r = 1 — 1 2 . 
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Alternate Solution 

It it not necessary to physically exchange equations during pivoting. We could accom- 
plish Gauss elimination just as well by keeping the equations in place. The elimination 
would then proceed as follows (for the sake of brevity, we skip repeating the details of 
choosing the pivot equation): 


II 

r "j=" 

, < 

2 

-2 

-2 6 

4 3 

16 

0 


1 

1 

t—* 

8 4 

-1 

i-H 

1 



0 2 9 

16 

II 

42 

< , 

-2 4 3 

0 6 5/2 

0 

-1 



0 

0 

49/6 

49/3" 

-2 

4 

3 

0 

0 

6 

5/2 

-1 


But now the back substitution phase is a little more involved, since the order in which 
the equations must be solved has become scrambled. In hand computations this is 
not a problem, because we can determine the order by inspection. Unfortunately, 
“by inspection” does not work on a computer. To overcome this difficulty, we have 
to maintain an integer array p that keeps track of the row permutations during the 
elimination phase. The contents of p indicate the order in which the pivot rows were 
chosen. In this example, we would have at the end of Gauss elimination 


P = 


2 

3 

1 


showing that row 2 was the pivot row in the first elimination pass, followed by row 3 
in the second pass. The equations are solved by back substitution in the reverse 
order: Eq. (1) is solved first for x 3 , then Eq. (3) is solved for x 2 , and finally Eq. (2) 
yields x 3 . 

By dispensing with swapping of equations, the scheme outlined above would 
probably resuit in a faster (and more complex) algorithm than gaussPivot, but the 
number of equations would have to be quite large before the difference becomes 
noticeable. 
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PROBLEM SET 2.2 

1. Solve the equations Ax = b by utilizing Doolittle’s decomposition, where 


3 

-3 

3" 


9" 

-3 

5 

1 

b = 

-7 

3 

1 

5 


12 


2. Use Doolittle’s decomposition to solve Ax = b, where 


' 4 

8 

20“ 


24~ 

8 

13 

16 

b = 

18 

20 

16 

-91 


-119 


3. Determine L and D that resuit from Doolittle’s decomposition of the symmetric 
matrix 



‘ 2 

-2 

0 

0 

0 


-2 

5 

-6 

0 

0 

A = 

0 

-6 

16 

12 

0 


0 

0 

12 

39 

-6 


0 

0 

0 

-6 

14 


4. Solve the tridiagonal equations Ax = b by Doolittle’s decomposition method, 
where 


' 6 

2 

0 

0 

0' 


' 2" 

-1 

7 

2 

0 

0 


-3 

0 

-2 

8 

2 

0 

b = 

4 

0 

0 

3 

7 

-2 


-3 

0 

0 

0 

3 

5. 


1 _ 


5. Use Gauss elimination with scaled row pivoting to solve 


‘4-2 l" 


Xl 


2 

-2 1 -1 


x 2 

= 

-1 

-2 3 6 


_ X 3_ 


0 


6. Solve Ax = b by Gauss elimination with scaled row pivoting, where 



2.34 

-4.10 

1.78“ 


0.02' 

A = 

-1.98 

3.47 

-2.22 

b = 

-0.73 


2.36 

-15.17 

6.81 


-6.63 
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7. Solve the equations 


" 2-1 0 0 ' 


"*r 


'l' 

0 0-11 




0 

0-1 2-1 




0 

-1 2 -1 0 _ 


X 4 


_0_ 


by Gauss elimination with scaled row pivoting. 

8. ■ Solve the equations 


0 2 5 -1' 


~x{ 


"-3" 

2 13 0 


x 2 


3 

-2-1 3 1 


x 3 


-2 

3 3-1 2_ 


_x 4 _ 


5_ 


9. ■ Solve the symmetric, tridiagonal equations 

4jq - x 2 = 9 

— Xi-i + 4xi - jtj+i = 5, i = 2,..., n— 1 
-x„_i + 4x„ = 5 


with n = 10. 

10. ■ Solve the equations Ax = b, where 



1.3174 

2.7250 

2.7250 

1.7181' 


"8.4855" 

0.4002 

0.8278 

1.2272 

2.5322 

V» — 

4.9874 

0.8218 

1.5608 

0.3629 

2.9210 

D — 

5.6665 

1.9664 

2.0011 

0.6532 

1.9945_ 


_6.6152_ 


11. ■ Solve the equations 


" 10 

-2 

-1 

2 

3 

1 

-4 

7" 


Xl 


O" 

5 

11 

3 

10 

-3 

3 

3 

-4 


x 2 


12 

7 

12 

1 

5 

3 

-12 

2 

3 


X 3 


-5 

8 

7 

-2 

1 

3 

2 

2 

4 


x 4 


3 

2 

-15 

-1 

1 

4 

-1 

8 

3 




-25 

4 

2 

9 

1 

12 

-1 

4 

1 


Xe 


-26 

-1 

4 

-7 

-1 

1 

1 

-1 

-3 


Xl 


9 

-1 

3 

4 

1 

3 

-4 

7 

6 


Xr 


-7 


12. ■ The system shown in Fig. (a) consists of // linear springs that support n masses. 
The spring stiffnesses are denoted by ki, the weights of the masses are W, and 
Xj are the displacements of the masses (measured from the positions where the 
springs are undeformed). The so-called displacementformulation is obtained by 
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writing the equilibrium equation of each mass and substituting Fi = k t (jq + 1 — x,) 
for the spring forces. The resuit is the symmetric, tridiagonal set of equations 

(h + k 2 )x i - k 2 x 2 = Wi 

-kiXi-i + (k + k i+ i)x t - ki+iXt+i = W ir i = 2, 3 ,. .., n- 1 
—k n x n —\ + k n x n = W n 

Write a program that solves these equations for given values of n, k and W. Run 
the program with n = 5 and 


ki = k 2 = k 3 = 10 N/mm fcj = h, = 5 N/mm 

W 1 = W 3 =W 5 = 100 N W 2 =W 4 = 50N 




(b) 


13. ■ The displacement formulation for the mass-spring system shown in Fig. (b) 
results in the following equilibrium equations of the masses: 


k\ + k 2 + k 3 + k$ —k 3 —h, 


Xl 


~Wi~ 

— k 3 k 3 T A 4 — /q 


x 2 

= 

w 2 

— ks —ki 1<4 + A 5 


_x 3 _ 


_w 3 _ 


where fc,- are the spring stiffnesses, W, represent the weights of the masses, and 
Xj are the displacements of the masses from the undeformed configuration of 
the system. Write a program that solves these equations, given k and W. Use the 
program to find the displacements if 


k\ = k 3 = k\ = k k 2 = ka = 2k 
W 1 = W 3 = 2W W 2 = W 
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14. ■ 



The displacement formulation for a plane truss is similar to that of a mass-spring 
System. The differences are: (1) the stiffnesses of the members are Iq = ( EA/L)i, 
where E is the modulus of elasticity, A represenLs die cross-sectional area and Lis 
the length of the member; (2) there are two components of displacement at each 
joint. For the statically indeterminate truss shown the displacement formulation 
yields the symmetric equations Ku = p, where 



27.58 

7.004 

-7.004 

0.0000 

0.0000' 



7.004 

29.57 

-5.253 

0.0000 

-24.32 


K = 

-7.004 

-5.253 

29.57 

0.0000 

0.0000 

MN/m 


0.0000 

0.0000 

0.0000 

27.58 

-7.004 



. 0.0000 

-24.32 

0.0000 

-7.004 

29.57. 




p = [o 

0 0 0 

-45] T kN 


Determine the 

displacements Ui of the joints. 



15. ■ 


12 kN 



In th e force formulation of a truss, the unknowns are the member forces P,-. For 
the statically determinate truss shown, the equilibrium equations of the joints 


"-1 

1 

-1/V2 

0 

0 

0' 


>r 


" 0“ 

0 

0 

1/V2 

1 

0 

0 


Pl 


18 

0 

-1 

0 

0 

-1/V2 

0 


p 3 


0 

0 

0 

0 

0 

1/V2 

0 


Pa 


12 

0 

0 

0 

0 

1/V2 

1 


p 5 


0 

0 

0 

0 

-1 

-1/V2 

0 


Pe 


0 


are: 
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where the units of P, are kN. (a) Solve the equations as they are with a computer 
program. (b) Rearrange the rows and columns so as to obtain a lower triangular 
coefficient matrix, and then solve the equations by back substitution using a 
calculator. 



The force formulation of the symmetric truss shown results in the joint equilib- 
rium equations 


"c 1 0 0 0' 


~PT 


'0' 

0 s 0 0 1 


P 2 


0 

0 0 2s 0 0 


Ps 

= 

1 

0 — c c 1 0 


P 4 


0 

O 

O 

O 


,h. 


_ 0 _ 


where s = sin /9, c = cos 6 and P, are the unknown forces. Write a program that 
computes the forces, given the angle 6. Run the program with 6 = 53°. 



The electrical network shown can be viewed as consisting of three loops. Apply- 
ing Kirchoffs law (^vohage drops = ^vollage sources) to each loop yields the 
following equations for the loop currents i\, k and k\ 

5i\ + 15(4 - 4) = 220 V 
P(4 — 4) + 54 + 104 = 0 
204 + P(4 — 4) + 15(4 — 4) = 0 


Compute the three loop currents for R = 5,10 and 20 fi. 
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18. ■ 



+120 V 

a 

io 


a 

in 


Determine the loop currents i\ to 4 in the electrical network shown. 

19. ■ Consider the n simultaneous equations Ax = b, where 

n— 1 

Aij = (i + j) 2 bi = ^2 Aij, i = 0,1 ,..., n— 1, j = 0,1,..., n— 1 

i=o 

The solution is x = |^1 1 ■■■ lj. Write a program that solves these equations 

for any given n (pivoting is recommended). Run the program with n = 2, 3 and 4, 
and comment on the results. 


*2.6 Matrix Inversion 

Computing the inverse of a matrix and solving simultaneous equations are related 
tasks. The most economical way to invert an n x n matrix A is to solve the equations 

AX = I (2.33) 

where I is the n x nidentity matrix. The solutionX, also of size n x n, will be the inverse 
of A. The proof is simple: after we premultiply both sides of Eq. (2.33) byA -1 we have 
A -1 A X = A -1 I, which reduces to X = A -1 . 

Inversion of large matrices should be avoided whenever possible due its high 
cost. As seen from Eq. (2.33), inversion of A is equivalent to solving Ax,— b, : with 
i = 1,2,..., n, where b, is the z'th column of I. If LU decomposition is employed in 
the solution, the solution phase (forward and back substitution) must be repeated n 
times, once for each b,. Since the cost of computation is proportional to n’ for the 
decomposition phase and n 2 for each vector of the solution phase, the cost of inversion 
is considerably more expensive than the solution of Ax = b (single constant vector b). 

Matrix inversion has another serious drawback—a banded matrix loses its struc¬ 
ture during inversion. In other words, if A is banded or otherwise sparse, then A -1 is 
fully populated. However, the inverse of a triangular matrix remains triangular. 
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EXAMPLE 2.13 

Write a function that inverts a matrix using LU decomposition with pivoting. Test the 
function by inverting 


0.6 

-0.4 

1.0 

-0.3 

0.2 

0.5 

0.6 

-1.0 

0.5 


Solution The function matlnv listed below uses the decomposition and solution 
procedures in the module LUpivot. 


#!/usr/bin/python 
## example2_13 

from numarray import array,identity, matrixmultiply 
from LUpivot import * 


def matlnv(a): 

n = len(a[0]) 
alnv = identity(n)*1.0 
a,seq = LUdecomp(a) 
for i in range(n): 

alnv[:,i] = LUsolve(a,alnv[:,i],seq) 
return alnv 

a = array([[ 0.6, -0.4, 1.0],\ 

[-0.3, 0.2, 0.5] , \ 

[ 0.6, -1.0, 0.5]]) 

aOrig = a.copyO # Save original [a] 

alnv = matlnv(a) # Invert [a] (original [a] is destroyed) 
print ’’ \nalnv =\n’’,alnv 

print ’’\nCheck: a*alnv =\n’’, matrixmultiply(aOrig,alnv) 
raw_input(’’ \nPress return to exit’’) 

The output is 

alnv = 

[[ 1.66666667 
[ 1.25 
[0.5 

Check: a*alnv = 

[[ 1.00000000e+00 -4.44089210e-16 -1.11022302e-16] 

[ 0.00000000e+00 1.00000000e+00 5.55111512e-17] 

[ 0.00000000e+00 -3.33066907e-16 1.00000000e+00]] 


- 2.22222222 - 1 . 11111111 ] 
-0.83333333 -1.66666667] 
1 . 0 . ]] 
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EXAMPLE 2.14 

Invert the matrix 



2 

-1 

0 

0 

0 

0 

1 

2 

-1 

0 

0 

0 

0 

-1 

2 

-1 

0 

0 

0 

0 

-1 

2 

-1 

0 

0 

0 

0 

-1 

2 

-1 

0 

0 

0 

0 

-1 

5 


Solution Since the matrix is tridiagonal, we solve AX = I using the functions in the 
module LUdecomp 3 (LU decomposition of tridiagonal matrices). 

#!/usr/bin/python 
## example2_14 

from numarray import array,ones,identity,Float64 
from LUdecomp3 import * 


n = 6 

d = ones((n))*2.0 

e = ones((n-1))*(-1.0) 

c = e.copyO 

d[n-l] = 5.0 

alnv = identity(n)*1.0 

c,d,e = LUdecomp3(c,d,e) 

for i in range(n): 

alnv[:,i] = LUsolve3(c,d,e,alnv[:,i]) 
print ’’ \nThe inverse matrix is:\n’’,alnv 
raw_input(’’ \nPress return to exit’’) 

Running the program results in the following output: 


The inverse matrix is: 


[[ 

0. 

. 84 

0. 

. 68 

0. 

. 52 

0. 

. 36 

0. 

. 2 

0. 

.04] 

[ 

0. 

.68 

1. 

. 36 

1. 

.04 

0. 

. 72 

0. 

.4 

0. 

.08] 

[ 

0. 

. 52 

1. 

.04 

1. 

. 56 

1. 

.08 

0. 

.6 

0. 

.12] 

[ 

0. 

. 36 

0. 

. 72 

1. 

.08 

1. 

.44 

0. 

.8 

0. 

.16] 

[ 

0. 

. 2 

0. 

.4 

0. 

.6 

0. 

.8 

1. 


0. 

■ 2 ] 

[ 

0. 

.04 

0. 

.08 

0. 

. 12 

0. 

. 16 

0. 

. 2 

0. 

.24]]] 


Note that A is tridiagonal, whereas A 1 is fully populated. 
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*2.7 Iterative Methods 
Introduction 

So far, we have discussed only direct methods of solution. The common characteristic 
of these methods is that they compute the solution with a finite number of operations. 
Moreover, if the computer were capable of infinite precision (no roundoff errors), the 
solution would be exact. 

Iterative, or indirect methods, start with an initial guess of the solution x and then 
repeatedly improve the solution until the change in x becomes negligible. Since the 
required number of iterations can be large, the indirect methods are, in general, slower 
than their direct counterparts. However, iterative methods do have the following ad- 
vantages that make them attractive for certain problems: 

1. It is feasible to store only the nonzero elements of the coefficient matrix. This 
makes it possible to deal with very large matrices that are sparse, but not neces- 
sarily banded. In many problems, there is no need to store the coefficient matrix 
at ali. 

2. Iterative procedures are self-correcting, meaning that roundoff errors (or even 
arithmetic mistakes) in one iterative cycle are corrected in subsequent cycles. 

A serious drawback of iterative methods is that they do not always converge to the 
solution. It can be shown that convergence is guaranteed only if the coefficient matrix 
is diagonally dominant. The initial guess for x plays no role in determining whether 
convergence takes place—if the procedure converges for one starting vector, it would 
do so for any starting vector. The initial guess affects only the number of iterations 
that are required for convergence. 


Gauss-Seidel Method 

The equations Ax = b are in scalar notation 


n 

y, AijXj = bi, 1=1,2,...,« 

7=1 

Extracting the term containing x, from the summation sign yields 


n 

Auxi + y AijXj = b it i = 1 . 2 ,...,« 
7=1 
7 ¥* 




86 


Systems of Linear Algebraic Equations 


Solving for x,-, we get 



i = 1,2 ,..., n 


The last equation suggests the following iterative scheme 



(2.34) 


We start by choosing the starting vector x. If a good guess for the solution is not 
available, x can be chosen randomly. Equation (2.34) is then used to recompute each 
element of x, always using the latest available values of xj . This completes one iteration 
cycle. The procedure is repeated until the changes in x between successive iteration 
cycles become sufficiently small. 

Convergence of the Gauss-Seidel method can be improved by a technique known 
as relaxation. The idea is to take the newvalue of x,- as aweighted average of its previous 
value and the value predicted by Eq. (2.34). The corresponding iterative formula is 



(2.35) 


where the weight <o is called the relaxation factor. It can be seen that i 1' o> = 1, no re¬ 
laxation takes place, since Eqs. (2.34) and (2.35) produce the same resuit. If w < 1, Eq. 

(2.35) represents interpolation between the old x; and the value given by Eq. (2.34). 
This is called underrelaxation. In cases where a> > 1, we have extrapolation, or over- 
relaxation. 

There is no practical method of determining the optimal value of o> beforehand; 
however, a good estimate can be computed during run time. Let Ax (fc) = |x (fc-1) — x (fc) | 
be the magnitude of the change in x during the k th iteration (carried out without 
relaxation, i.e., with o> = 1). If Ar is sufficiently large (say k > 5), it can be shown 4 that 
an approximation of the optimal value of o> is 


2 


^opt 


(2.36) 


l + Jl- (Ax' fc +PVAx' fc >) 1/p 


where p is a positive integer. 


4 See, for example, Terrence J. Akai, Applied Numerical Methods for Engineers, Iohn Wiley & Sons 
(1994), p. 100. 
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The essential elements of a Gauss-Seidel algorithm with relaxation are: 

1. Carry out Ic iterations with u>= 1 (fc= 10 is reasonable). After the fcth iteration 
record Ax (fc) . 

2. Perform an additional piterations and record Ax (k+/fl for the last iteration. 

3. Perform all subsequent iterations with co = &> opt , where « op t is computed from 
Eq. (2.36). 

■ gaussSeidel 

The function gaussSeidel is an implementation of the Gauss-Seidel method with 
relaxation. It automatically computes w opt from Eq. (2.36) using k= 10 and p= 1. 
The user must provide the function iterEqs that computes the improved x from the 
iterative formulas in Eq. (2.35)—see Example 2.17. The function returns the solution 
vector x, the number of iterations carried out and the value of <w op t used. 

## module gaussSeidel 

’’’ x,numlter,omega = gaussSeidel(iterEqs,x,tol = 1.0e-9) 
Gauss-Seidel method for solving [A]{x> = {b}. 

The matrix [A] should be sparse. User must supply the 
function iterEqs(x,omega) that returns the improved {x>, 
given the current {x} ('omega’ is the relaxation factor). 

from numarray import dot 
from math import sqrt 

def gaussSeidel(iterEqs,x,tol = 1.0e-9): 
omega = 1.0 
k = 10 

p = 1 

for i in range(l,501): 
xOld = x.copyO 
x = iterEqs(x,omega) 
dx = sqrt(dot(x-xOld,x-xOld)) 
if dx < tol: return x,i,omega 
# Compute of relaxation factor after k+p iterations 
if i == k: dxl = dx 
if i == k + p: 
dx2 = dx 

omega = 2.0/(1.0 + sqrt(1.0 - (dx2/dxl)* *(1.0/p))) 
print 'Gauss-Seidel failed to converge’ 
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Conjugate Gradient Method 

Consider the problem of finding the vector x that minimizes the scalar function 

/(x) = ^x r Ax — b r x (2.37) 

where the matrix A is symmetric and positive definite. Because /(x) is minimized 
when its gradient V / = Ax — b is zero, we see that minimization is equivalent to 
solving 


Ax = b (2.38) 

Gradient methods accomplish the minimization by iteration, starting with an 
initial vector xq. Each iterative cycle k computes a refined solution 


Xfc+i — x k -\-ot k s k (2.39) 

The step length a k is chosen so that x^ +1 minimizes J'[x k+ 1 ) in the search direction s k . 
That is, Xfc + i must satisfy Eq. (2.38): 

A(x* + a k s fc ) = b (a) 


Introducing the residual 


r k = b- Ax k 


(2.40) 


Eq. (a) becomes uAs k = r k . Premultiplying both sides by s k and solving for a k , we 
obtain 


a k = 


s l r k 

s k As k 


(2.41) 


We are stili left with the problem of determining the search direction s k . Intuition 
telis us to choose s k = - A/ f = r k , since this is the direction of the largest negative 
change in /(x). The resulting procedure is known as the method of steepest descent. 
It is not a popular algorithm since its convergence can be slow. The more efficient 
conjugate gradient method uses the search direction 

Sfc+i = rj; + i + fi k s k (2.42) 

The constant f) k is chosen so that the two successive search directions are conjugate 
to each other, meaning 


s k+1 As k = 0 


(b) 
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The great attraction of conjugate gradients is that minimization in one conjugate 
direction does not undo previous minimizations (minimizations do not interfere with 
one another). 

Substituting s* + i from Eq. (2.42) into Eq. (b), we get 

( r ifc+l + Pk s k) ASA: = 0 


which yields 


Pk = 


*I+ i As t- 

s[As fc 


Eiere is the outline of the conjugate gradient algorithm: 


(2.43) 


• Choose xo (any vector will do, but one close to solution results in fewer iterations) 

• r 0 «- b — Axo 

• s 0 <— r 0 (lacking a previous search direction, choose the direction of steepest 
descent) 

• do with k= 0,1, 2 ,... 


Otk <- 


s l r k 

SfcAs k 


Xfc+i <- Xfc + a k s k 
ffc+i •<- b - Ax fc+ i 

if Iht+il < e ex it loop (e is the error tolerance) 

r fc +1 A S fc 
Pk * t » 

sf-As k 

Sfc+l <— + /Sj-Sfc 

end do 


It can be shown that the residual vectors ri, r 2 , r 3 ,... produced by the algorithm 
aremutuallyorthogonal;thatis,rj • r j = 0, i / /. Novvsuppose IluUwehavecarried ouL 
enough iterations to have computed the whole set of n residual vectors. The residual 
resulting from the next iteration must be the null vector (r„ + 1 = 0), indicating that the 
solution has been obtained. It thus appears that the conjugate gradient algorithm is 
not an iterative method at ali, since it reaches the exact solution after «computational 
cycles. In practice, however, convergence is usually achieved inless than «iterations. 

The conjugate gradient method is not competitive with direct methods in the 
solution of small sets of equations. Its strength lies in the handling of large, sparse 
Systems (where most elements of A are zero). It is important to note that A enters the 
algorithm only through its multiplication by a vector; that is, in the form Av, where v is 
a vector (either x^+i or s k ). If A is sparse, it is possible to write an efficient subroutine for 
the multiplication and pass it, rather than A itself, to the conjugate gradient algorithm. 
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■ conjGrad 

The function conjGrad shown below implements the conjugate gradient algorithm. 
The maximum allowable number of iterations is set to n (the number of unknowns). 
Note that conjGrad calls the function Av which returns the product Av. This func¬ 
tion must be supplied by the user (see Example 2.18). We must also supply the starting 
vector xo and the constant (right-hand-side) vector b. The function returns the solu- 
tion vector x and the number of iterations: 

## module conjGrad 

’’’ x, numlter = conjGrad(Av,x,b,tol=l.Oe-9) 

Conjugate gradient method for solving [A]{x> = {b}. 

The matrix [A] should be sparse. User must supply 
the function Av(v) that returns the vector [A]{v>. 

from numarray import dot 
from math import sqrt 

def conjGradCAv,x,b,tol=l.0e-9): 
n = len(b) 
r = b - Av(x) 
s = r.copyQ 
for i in range(n): 
u = Av(s) 

alpha = dot(s,r)/dot(s,u) 
x = x + alpha*s 
r = b - Av(x) 
if(sqrt(dot(r,r))) < tol: 

break 
else : 

beta = -dot(r,u)/dot(s,u) 
s = r + beta*s 
return x,i 

EXAMPLE 2.15 

Solve the equations 


' 4-1 r 


Xl 


~ 12" 

CN 

1 

i—l 

i 


x 2 

= 

-1 

1 -2 4 


_x 3 _ 


5 


by the Gauss-Seidel method without relaxation. 
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Solution With the given data, the iteration formulas in Eq. (2.34) become 

1 

xi = - (12 + x 2 - x 3 ) 

4 

1 

x 2 — — (— 1 + X\ + 2x 3 ) 

1 

X3 = - (5 - Xi + 2x 2 ) 

Choosing the starting values X\ = x 2 = x 3 = 0, we have for the first iteration 

Xi = ^ (12 + 0 - 0) = 3 
1 

x 2 = - [-1 + 3 + 2(0)] = 0.5 
4 

x 3 = i [5 - 3 + 2(0.5)] = 0.75 

The second iteration yields 

1 

Xi = - (12 + 0.5 - 0.75) = 2.9375 

1 

x 2 = — [-1 + 2.9375 + 2(0.75)] = 0.859 38 
4 

1 

x 3 = - [5 - 2.9375 + 2(0.859 38)] = 0.945 31 
and the third iteration results in 

JCi = ^ (12 + 0.85938 - 0.94531) = 2.978 52 
1 

x 2 = — [-1 + 2.97852 + 2(0 .94531)] = 0.967 29 
4 

x 3 = ^ [5 - 2.97852 + 2(0.96729)] = 0.989 02 

After five more iterations the results would agree with the exact solution x 3 = 3, 
x 2 = x 3 = 1 within five decimal places. 

EXAMPLE 2.16 

Solve the equations in Example 2.15 by the conjugate gradient method. 

Solution The conjugate gradient method should converge after three iterations. 
Choosing again for the starting vector 

xo=[o 0 of 
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the computations outlined in the text proceed as follows: 
First iteration 



"l2‘ 


' 4-1 r 


'o' 


’ 12" 

II 

1 

1 

-Q 

II 

o 

u 

-1 

- 

-1 4 -2 


0 

= 

-1 


5 


1 

I — 1 

1 

ro 

i 


0 


5 


s 0 = r 0 = 


12 

-1 

5 


'4-1 l" 


" 12" 


54“ 

-1 4 -2 


-1 

= 

-26 

1 -2 4 


5 


34 


s o r o 

SqASq 


12 2 + (—l) 2 + 5 2 
12(54) + (—1)(—26) + 5(34) 


0.20142 


Xi = Xo + aoSo = 

Second iteration 


o" 


" 12" 


2.41 704’ 

0 

+ 0.20142 

-1 

= 

-0.20142 

0 


5 


1.00710 


’ 12" 


'4-1 l" 


2.417 04" 


1.123 32’ 

-1 

- 

-1 4 -2 


-0.20142 

= 

4.236 92 

5 


1 

I— 1 

1 

ro 

i_ 


1.00710 


-1.848 28 




r[Aso 

SqAsq 


1.123 32(54) + 4.236 92(—26) - 1.848 28(34) 
12(54)+ (-l)(-26) +5(34) 


0.133107 


’ 1.123 32 ’ 


’ 12’ 


2.720 76’ 

4.23692 

+ 0.133107 

-1 

= 

4.103 80 

-1.84828 


5 


-1.182 68 


'4-1 l’ 


2.720 76’ 


5.59656’ 

-1 4 -2 


4.103 80 

= 

16.059 80 

1 -2 4 


-1.182 68 


-10.21760 
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s^Asi 

2.72076(1.123 32) + 4.103 80(4.23692) + (-1.182 68)(-1.848 28) 
"~ 2.720 76(5.596 56) + 4.103 80(16.059 80) + (-1.182 68)(-10.21760) 
= 0.24276 


2.417 04" 


2.720 76" 


"3.077 53" 

-0.20142 

+ 0.24276 

4.103 80 

= 

0.794 82 

1.00710 


-1.182 68 


0.719 99 


Third iteration 


" 12 " 


"4-1 l’ 


"3.077 53" 


—0.235 29 _ 

-1 

- 

-1 4 -2 


0.794 82 

= 

0.338 23 

5 


1 

CN 

1 

i-H 

_ 1 


0.719 99 


0.632 15 


(-0.235 29) (5.596 56) + 0.338 23(16.059 80) + 0.632 15(-10.217 60) 
"~ "~ 2.720 76(5.596 56) + 4.103 80(16.059 80) + (-1.182 68 ) (-10.217 60) 

= 0.0251452 


—0.235 29" 


2.720 76" 


"—0.166 876" 

0.33823 

+ 0.025 1452 

4.103 80 

= 

0.441421 

0.63215 


-1.182 68 


0.602411 



4 -1 1 


—0.166 876 

-0.506 514 

As 2 = 

-1 4 -2 


0.441421 

= 0.727 738 


1 -2 4 


0.602 411 

1.359 930 


= r 2 S 2 
sjAs 2 

(-0.235 29) (-0.166 876) + 0.338 23(0.441 421) + 0.632 15(0.602 411) 

"~ (-0.166876)(-0.506514) + 0.441421(0.727738) + 0.602411(1.359930) 
= 0.464 80 


"3.077 53" 


"—0.166876" 


"2.999 97" 

0.794 82 

+ 0.464 80 

0.441421 

= 

0.999 99 

0.719 99 


0.602 411 


0.999 99 


The solution X 3 is correct to almost live decimal places. The small discrepancy is 
caused by roundoff errors in the computations. 
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EXAMPLE 2.17 

Write a computer program to solve the following n simultaneous equations by the 
Gauss-Seidel method with relaxation (the program should work with any value of n) 5 . 


2 

-1 

0 

0 .. 

0 

0 

0 

1 ' 


" *1 ' 


' 0 " 

-1 

2 

-1 

0 .. 

0 

0 

0 

0 




0 

0 

-1 

2 

-1 .. 

0 

0 

0 

0 




0 

0 

0 

0 

0 .. 

. -1 

2 

-1 

0 


Xn- 2 


0 

0 

0 

0 

0 .. 

0 

-1 

2 

-1 


%n— 1 


0 

1 

0 

0 

0 .. 

0 

0 

-1 

2 _ 




_ 1 _ 


Run the program with n = 20. The exact solution can be shown to be x,- 
i = 1 , 2 ,...,«. 

Solution In this case the iterative formulas in Eq. (2.35) are 
X\ = w{x 2 - x„)/2 + (1 - w)xi 

Xi = <w(x;_i + Xj+i)/2 + (1 - co)Xi, i = 2, 3,...,«- 1 
x n — cv[ 1 X\ + x h _x)/2 (1 oj)x n 

These formulas are evaluated in the function iterEqs. 

#!/usr/bin/python 
## example2_17 

from numarray import zeros,Float64 
from gaussSeidel import * 

def iterEqs(x,omega): 
n = len(x) 

x[0] = omega*(x[l] - x[n-l])/2.0 + (1.0 - omega)*x[0] 
for i in range(1,n-1): 

x[i] = omega*(x[i-l] + x[i+l])/2.0 + (1.0 - omega)*x[i] 
x[n-l] = omega*(1.0 - x[0] + x[n-2])/2.0 \ 

+ (1.0 - omega)*x[n-l] 
return x 

n = eval(raw_input(’’Number of equations ==> ’’)) 
x = zeros((n),type=Float64) 
x,numlter,omega = gaussSeidel(iterEqs,x) 

5 Equations of this form are called cyclic tridiagonal. They occur in the finite difference formulation 
of second-order differentia! equations with periodic boundary conditions. 


= —n/ 4 + i/ 2, 


(a) 
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print ’’\nNumber of iterations ='’,numlter 
print ’’ \nRelaxation factor =’’,omega 
print ’’ \nThe solution is:\n’’,x 
raw_input(’’ \nPress return to exit’’) 

The output from the program is: 

Number of equations ==> 20 


Number of iterations = 259 

Relaxation factor = 1.70545231071 


The solution is : 
[-4.50000000e+00 
-2.50000000e+00 
-4.99999998e-01 
1.50000000e+00 
3.50000000e+00 


-4.00000000e+00 
-2.00000000e+00 
2.14046747e-09 
2.00000000e+00 
4.00000000e+00 


-3.50000000e+00 
-1.50000000e+00 
5.00000002e-01 
2.50000000e+00 
4.50000000e+00 


-3.00000000e+00 
-9.99999997e-01 
1.00000000e+00 
3.00000000e+00 
5.00000000e+00] 


The convergence is very slow, because the coefficient matrix lacks diagonal 
dominance—substituting the elements of Ainto Eq. (2.30) produces an equality rather 
than the desired inequality. Ifwe were to change each diagonal term of the coefficient 
matrix from 2 to 4, A would be diagonally dominant and the solution would converge 
in about 20 iterations. 

EXAMPLE 2.18 

Solve Example 2.17 with the conjugate gradient method, also using n = 20. 

Solution The program shown below utilizes the function conjGrad. The solution 
vector x is initialized to zero in the program, which also sets up the constant vector b. 
The function Av( v) returns the product Av, where A is the coefficient matrix and v is 
a vector. For the given A, the components of the vector Av are 


(Av)i = 2vi - V 2 + v n 

(Av )i = -i/,_i + 2 Vi - v i+ i, i = 2, 3,..., n- 1 
(Av)„ = -v n -i +2v n + Vi 
which are evaluated by the function Av(v). 

#!/usr/bin/python 
## example2_18 
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from numarray import zeros,Float64,sqrt 
from conjGrad import * 

def Ax(v): 

n = len(v) 

Ax = zeros((n),type=Float64) 

Ax[0] = 2.0*v[0] - v[l] + v[n-l] 

Ax[l:n-1] = -v[0:n-2] + 2.0*v[l:n-l] - v[2:n] 

Ax[n-1] = -v[n-2] + 2.0*v[n-l] + v[0] 
return Ax 

n = eval(raw_input(’’Number of equations ==> ’’)) 
b = zeros((n),type=Float64) 
b[n-1] = 1.0 

x = zeros((n),type=Float64) 

x.numlter = conjGrad(Ax,x,b) 

print ’’ \nThe solution is:\n’’,x 

print ’’ \nNumber of iterations =’’,numlter 

raw_input('’ \nPress return to exit’’) 

Running the program results in 

Number of equations ==> 20 

The solution is : 

[-4.5 -4. -3.5 -3. -2.5 -2. -1.5 -1. -0.5 0. 0.5 1. 1.5 

2. 2.5 3. 3.5 4. 4.5 5. ] 

Number of iterations = 9 

Note that convergence was reached in only 9 iterations, whereas 259 iterations 
were required in the Gauss-Seidel method. 

PROBLEM SET 2.3 



1. Let 
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(note that B is obtained by interchanging the first two rows of A). Knowing that 


A" 1 = 

determine B -1 . 

2. Invert the triangular matrices 


A = 


3. Invert the triangular matrix 


0.5 

0 

0.25 

0.3 

0.4 

0.45 

-0.1 

0.2 

-0.15 


‘2 

4 

3" 


‘2 

0 

0 " 

0 

6 

5 

B = 

3 

4 

0 

0 

0 

2 


4 

5 

6 


A = 


4. Invert the following matrices: 


(a) A = 


1 1/2 1/4 1/8 


1 1/3 1/9 

0 1 1/4 

0 0 1 


"l 

2 

4 


4 

-1 

0 " 

1 

3 

9 

(b) B = 

-1 

4 

-1 

1 

4 

16 


0 

-1 

4 


5. Invert the matrix 


A = 


4 

-2 


-2 1 

1 -1 


1 -2 4 

6 . ■ Invert the following matrices with any method: 



5 

-3 

-1 

0 ' 


4 

-1 

0 

0 “ 


-2 

1 

1 

1 

B = 

-1 

4 

-1 

0 

A = 

3 

-5 

1 

2 

0 

-1 

4 

-1 



0 

8 

-4 

-3 


0 

0 

-1 

4_ 


7. ■ Invert the matrix with any method: 

1 


A = 


3 

2 -1 
3 2 

8 -1 

11 1 


-9 

6 

-3 

1 


6 

7 

15 

4 


-2 18 7 


and comment on the reliability of the resuit. 
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8 . ■ The joint displacements u of the plane truss in Prob. 14, Problem Set 2.2 are 
related to the applied joint forces p by 


Ku = p 


(a) 


where 


27.580 

7.004 

-7.004 

0.000 

0 .000' 

7.004 

29.570 

-5.253 

0.000 

-24.320 

-7.004 

-5.253 

29.570 

0.000 

0.000 

0.000 

0.000 

0.000 

27.580 

-7.004 

0.000 

-24.320 

0.000 

-7.004 

29.570 


MN/m 


is called the stiffness matrix of the truss. If Eq. (a) is inverted by multiplying each 
sidebyK -1 , weobtainu = K -1 p, where K -1 isknownas th e flexibility matrix. The 
physical meaning of the elements of the flexibility matrix is: 1 = displacements 
iM {i = 1, 2,..., 5) produced by the unit load pj = 1. Compute (a) the flexibility 
matrix of the truss; (b) the displacements of the joints due to the load /% = —45 
kN (the load shown in Prob. 14, Problem Set 2.2). 

9. ■ Invert the matrices 



' 3 

-7 

45 

21 " 


'1 

1 

1 

1 " 


12 

11 

10 

17 

B = 

1 

2 

2 

2 

A = 

6 

25 

-80 

-24 

2 

3 

4 

4 



17 

55 

-9 

7 


_4 

5 

6 

7 


10. ■ Write a program for inverting an nx n lovver triangular matrix. The inversion 
procedure should contain only forward substitution. Test the program by invert¬ 
ing the matrix 


36 0 0 0 

A _ 18 36 0 0 

9 12 36 0 

_ 5 4 9 36 

11. Use the Gauss-Seidel method to solve 


—2 5 9" 


X| 


l" 

7 1 1 



= 

6 

-3 7 -1 


_*3_ 


-26 
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12. Solve the following equations with the Gauss-Seidel method: 


"12-2 3 r 


"xi" 


' 0“ 

-2 15 6 -3 




0 

1 6 20 -4 




20 

0 -3 2 9_ 


X 4 


_ 0 _ 


13. Use the Gauss-Seidel method with relaxation to solve Ax = b, where 


4 

-1 

0 

0 ' 


"15" 

-1 

4 

-1 

0 

b = 

10 

0 

-1 

4 

-1 


10 

0 

0 

-1 

3_ 


_10_ 


Take x,- = /y/ Au as the starting vector and use co = 1.1 for the relaxation factor. 

14. Solve the equations 


"2-1 0" 


Xl 


"l" 

-1 2 -1 


x 2 

= 

1 

0 -1 1 


_X 3 _ 


1 


by the conjugate gradient method. Start with x = 0. 

15. Use the conjugate gradient method to solve 


‘3 0 -l" 


Xi 


4" 

0 4-2 



= 

10 

-1 -2 5 


_*3_ 


-10 


starting with x = 0. 

16. ■ Solve the simultaneous equations Ax = b and Bx = b by the Gauss-Seidel 
method with relaxation, where 



-8 

10 

10 

-8 

10 J 

3 

-2 

1 

0 

0 

0 

-2 

4 

-2 

1 

0 

0 

1 

-2 

4 

-2 

1 

0 

0 

1 

-2 

4 

-2 

1 

0 

0 

1 

-2 

4 

-2 

0 

0 

0 

1 

-2 
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3 

-2 

1 

0 

0 

1 

2 

4 

-2 

1 

0 

0 

1 

-2 

4 

-2 

1 

0 

0 

1 

-2 

4 

-2 

1 

0 

0 

1 

-2 

4 

-2 

1 

0 

0 

1 

-2 

3 


Note that A is not diagonally dominant, but that does not necessarily preclude 
convergence. 

17. ■ ModifytheprograminExample2.17 (Gauss-Seidel method) so that it will solve 
the following equations: 


4 

-1 

0 

0 •• 

0 

0 

0 

r 


’ Xi ' 


' o " 

-1 

4 

-1 

0 •• 

0 

0 

0 

0 




0 

0 

-1 

4 

-1 ■ ■ 

0 

0 

0 

0 




0 

0 

0 

0 

0 •• 

■ -1 

4 

-1 

0 


X n -2 


0 

0 

0 

0 

0 •• 

0 

-1 

4 

-1 


% n—l 


0 

1 

0 

0 

0 •• 

0 

0 

-1 

4 _ 


Xfi 


_ 100 _ 


Run the program with n = 20 and compare the number of iterations with Example 
2.17. 

18. ■ Modify the program in Example 2.18 to solve the equations in Prob. 17 by the 
conjugate gradient method. Run the program with n = 20. 


7=0° 


1 

2 

3 


4 

5 

6 


7 

8 

9 







T= 200° 


T= 100° 


The edges of the square piate are kept at the temperatures shown. Assuming 
steady-state heat conduction, the differential equation governing the temperature 
T in the interior is 


d 2 r d 2 r 
Hf ~ 
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2.8 Other Methods 


If this equation is approximated by finite differences using the mesh shown, we 
obtain the following algebraic equations for temperatures at the mesh points: 


‘-4 

1 

0 

1 

0 

0 

0 

0 

o' 


~Ti 


0" 

1 

-4 

1 

0 

1 

0 

0 

0 

0 


t 2 


0 

0 

1 

-4 

0 

0 

1 

0 

0 

0 


t 3 


100 

1 

0 

0 

-4 

1 

0 

1 

0 

0 


t 4 


0 

0 

1 

0 

1 

-4 

1 

0 

1 

0 


t 5 

= - 

0 

0 

0 

1 

0 

1 

-4 

0 

0 

1 


t 6 


100 

0 

0 

0 

1 

0 

0 

-4 

1 

0 


t 7 


200 

0 

0 

0 

0 

1 

0 

1 

-4 

1 


t 8 


200 

0 

0 

0 

0 

0 

1 

0 

1 

-4 


T 9 _ 


300 


Solve these equations with the conjugate gradient method. 


*2.8 Other Methods 

A matrix can be decomposed in numerous ways, some of which are generally useful, 
whereas others find use in special applications. The most important of the latter are 
the QR factorization and the singular value decomposition. 

The QR decomposition of a matrix A is 


A= QR 

where Q is an orthogonal matrix (recall that the matrix Q is orthogonal if Q -1 = Q r ) 
and R is an upper triangular matrix. Unlike LU factorization, QR decomposition does 
not require pivoting to sustain stability, but it does involve about twice as many op- 
erations. Due to its relative inefficiency, the QR factorization is not used as a general- 
purpose tool, but finds its niche in applications that put a premium on stabiliy (e.g., 
solution of eigenvalue problems). 

The singular value decomposition is useful in dealing with singular or ill- 
conditioned matrices. Here the factorization is 

A = UAV r 

where U and V are orthogonal matrices and 


A = 


Ai 0 0 

0 X 2 0 

0 0 ^3 
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is a diagonal matrix. The elements /,/ of A can be shown to be positive or zero. If 
A is symmetric and positive definite, then the /, ’s are the eigenvalues of A. A nice 
characteristic of the singular value decomposition is that it works even if A is singular 
or ill-conditioned. The conditioning of A can be diagnosed from magnitudes of the 
a ’ s : the matrix is singular if one or more of the /,’s are zero, and it is ill-conditioned if 
the condition number 


is very large. 


cond(A) — k max /k m in 
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Interpolation and Curve Fitting 


Given the n+ 1 data points (jt/, y,), i = 0,1,..., n, estimate y(x). 


3.1 Introduction 

Discrete data sets, or tables of the form 


Xq 

Xi 

X 2 



yo 

yi 

y2 


yn 


are commonly involved in technical calculations. The source of the data may be ex- 
perimental observations or numerical computations. There is a distinction between 
interpolation and curve fitting. In interpolation we construet a curve through the data 
points. In doing so, we make the implicit assumption that the data points are accurate 
and distinet. Curve fitting is applied to data that contain scatter (noise), usually due to 
measurement errors. Here we want to find a smooth curve that approximates the data 
in some sense. Thus the curve does not necessarily hit the data points. The difference 
between interpolation and curve fitting is illustrated in Fig. 3.1. 
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Interpolation and Curve Fitting 


3.2 Polynomial Interpolation 

Lagrange's Method 

The simplest form of an interpolant is a polynomial. It is always possible to construet 
a unique polynomial of degree n that passes through n+ 1 distinet data points. One 
means of obtaining this polynomial is the formula of Lagrange 


PnW = ^2 yitiW 

i =0 

where the subscript n denotes the degree of the polynomial and 


(3.1 a) 


ii (x) = 


X — Xq X — Xi X — Xi-i X — Xi + 1 x — x n 

Xi - Xq Xi - XI Xi - Xi-1 Xi - X i+ 1 X; - X„ 


n 


X — X 


j=o x ‘~ x j 
j¥=< 


i = 0,1,..., n 


(3.1b) 


are called the Cardinal functioris. 

For example, if n = 1, the interpolant is the straight line Pi(x) = y 0 f o M + yiiiM, 
where 


ioM = 


x — X\ 


h(x) = 


X- Xq 


Xq-Xi Xi — Xq 

With n = 2, interpolation is parabolic: P 2 (x) = yr>fo (x) + y\l\ (x) + y-J -2 (x), where now 

(X-Xi)(X-X 2 ) 


4W = 
fi(x) = 
l'Ax) = 


[Xq - Xi)(Xo - X 2 ) 
(X - Xq) (X - X 2 ) 
(Xi - Xo) (Xi - X 2 ) 
(X- Xq){X- Xi) 


(X 2 - Xq) (X 2 - Xi) 

The Cardinal functions are polynomials of degree «and have the property 

f 0 if i^ j , 

ltlX]) = \liU=j i=8i 


(3.2) 


where S y is the Kronecker delta. This property is illustrated in Fig. 3.2 for three-point 
interpolation (n = 2) with xo = 0, xi = 2 and x 2 = 3. 



Figure 3.2. Example of quadratic Cardinal functions. 
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3.2 Polynomial Interpolation 


To prove that the interpolating polynomial passes through the data points, we 
substitute x = xj into Eq. (3.Ia) and then utilize Eq. (3.2). The resuit is 


PniXj) — "y ' Vd i (Xj ) — y ' ViS j j — yj 
i =0 i =0 

It can be shown that the error in polynomial interpolation is 

/w - pm - /“«>«) 


(« + 1 )! 


(3.3) 


where f lies somewhere in the interval (jcq, x „); its value is otherwise unknown. It is 
instructive to note that the farther a data point is from x, the more it contributes to 
the error at x. 


NewtorTs Method 

Although Lagrange’s method is conceptually simple, it does not lend itself to an effi¬ 
cient algorithm. A better computational procedure is obtained with Newton’s method, 
where the interpolating polynomial is written in the form 

P„(x) = do + (x - Xbffii + (x - Xo)(x - Xi)«2 H-h (x - Xb)(x - Xi) • • ■ (x - x„_i )a n 

This polynomial lends itself to an efficient evaluation procedure. Consider, for 
example, four data points {n= 3). Here the interpolating polynomial is 

P 3 (X) = Oq + (x - Xo)«i + (x - x 0 )(x - Xi )a 2 + (x - x^)(x - xi)(x - x 2 )t% 

= flo + (x - Xo) {«1 + (x - Xi) [a 2 + (x - x 2 )o?]} 

which can be evaluated backwards with the following recurrence relations: 


Pq{x) = Cb, 

Pi (x) = fl 2 + (x - x 2 ) P 0 (x) 

P 2 (x) = fli + (x - Xi) Pi (x) 

P 3 (x) = flb + (x - x 0 ) P 2 (x) 

For arbitrary n vve have 

P Q {x) = a n P fc (x) = a n - k + (x-x„_ fc )P fc _i(x), k=l,2,...,n (3.4) 

Denoting the x-coordinate array of the data points by xData and the degree of the 
polynomial by n, we have the following algorithm for computing P n {x): 
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P = a[n] 

for k in range(1,n+1): 

p = a[n-k] + (x - xData[n-k])*p 

The coefficients of P„ are determined by forcing the polynomial to pass through 
each data point: y,- = P n [Xi), i = 0,1,..., n This yields the simultaneous equations 


y 0 = Oq 


y, = <3fl + (x, - x 0 )«i 

y 2 = ao +[x 2 -Xo)ai +(_x 2 -Xo){x 2 -X])a 2 (a) 


y n = ao + ( x n - xo)ai -{ -b (x„ - xo)(x„ - x 4 ) • • • (x„ - x n -i)a n 


Introducing the divided differences 


Vy 


V 2 y 

V 3 y 


yt ~ To 

Xi - Xo ’ 


i = 1,2 ,...,« 


X; - Xi 


i = 2,3,..., n 


v 2 y - v 2 y 2 

X; - x 2 


i = 3,4,... n 


(3.5) 


V ~ 

the solution of Eqs. (a) is 

flj) = y 0 « 1 = Vyi a 2 =V 2 y ••• a n = V n y n (3.6) 

If the coefficients are computed by hand, it is convenient to work with the format in 
Table 3.1 (shown for n = 4). 


Xo 

yo 





Xi 

yi 

Vy 




X 2 

y2 

Vy 

V 2 y 



X 3 

ys 

vy 3 

V 2 y 

V 3 y 3 


x 4 

T4 

Vy 4 

V 2 y 4 

v 3 y 4 

v 4 y 4 


Table3.1 
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3.2 Polynomial Interpolation 


The diagonal terms (y n , Vyi, V 2 y 2 , V 3 y 3 and V 4 y 4 ) in the table are the coefficients of 
the polynomial. If the data points are listed in a different order, the entries in the table 
will change, but the resultant polynomial will be the same—recall that a polynomial 
of degree ninterpolating n+ 1 distinet data points is unique. 

Machine computations can be carried out within a one-dimensional array a em- 
ploying the following algorithm (we use the notation m= n+ 1 = number of data 
points): 

a = yData.copyO 
for k in range(l,m): 

for i in range(k,m): 

a[i] = (a[i] - a[k-l])/(xData[i] - xData[k-l]) 

Initially, a contains the y-coordinates of the data, so that it is identical to the 
second column in Table 3.1. Each pass through the outer loop generates the entries in 
the next column, which overwrite the corresponding elements of a. Therefore, a ends 
up containing the diagonal terms of Table 3.1, i.e., the coefficients of the polynomial. 

■ newtonPoly 

This module contains the two functions required for interpolation by Newton’s 
method. Given the data point arrays xData and yData, the function coeffts re- 
turns the coefficient array a. After the coefficients are found, the interpolant P n [x) can 
be evaluated at any value of x with the function evalPoly. 

## module newtonPoly 

’’’ p = evalPoly(a,xData,x). 

Evaluates Newton’s polynomial p at x. The coefficient 
vector {a} can be computed by the function 'coeffts'. 

a = coeffts(xData,yData). 

Computes the coefficients of Newton’s polynomial. 

def evalPoly(a,xData,x): 

n = len(xData) - 1 # Degree of polynomial 

p = a[n] 

for k in range(1,n+1): 

p = a[n-k] + (x -xData[n-k])*p 
return p 

def coeffts(xData,yData): 

m = len(xData) # Number of data points 
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a = yData.copyO 
for k in range(l,m); 

a[k:m] = (a[k:m] - a[k-1])/(xData[k:m] - xData[k-l]) 
return a 


Neville's Method 


Newton's method of interpolation involves two steps: computation of the coefficients, 
followed by evaluation of the polynomial. This works well if the interpolation is carried 
out repeatedly at different values of x using the same polynomial. If only one point 
is to interpolated, a method that computes the interpolant in a single step, such as 
Neville’s algorithm, is a better choice. 

Let PklXi, Xj + \,..., Xj + k\ denote the polynomial of degree /cthat passes through the 
k+ 1 data points (xy). (x,-+ i, y;+ \) ,..., (Xi+fc, Yi+k)■ For a single data point, we have 


PoiXi] = y t 


(3.7) 


The interpolant based on two data points is 


P\ lXi, Xj+i] 


(X - Xj+i) P 0 [Xj\ + {Xj - X) Pp [Xj+i] 
Xi - x i+1 


It is easily verified that P\ [x,-, x,-+i ] passes through the two data points; that is, 
Pi [xt, Xj+i] = y t when x = x*, and P x [x,-, x i+1 ] = y i+ 1 when x = x i+l . 

The three-point interpolant is 


P 2 \Xj, x i+! , x ; - +2 ] 


(X - X i+2 ) P\ [Xi, X,-+i] + {Xj - X) Pi lXj+1 . x i+2 ] 
Xi - Xi+2 


To show that this interpolant does intersect the data points, we frrst substitute x = x,-, 
obtaining 


P 2 [Xi, Xi+1, X i+2 \ = P| [Xi, Xi+i] = yi 

Similarly, x = Xi+2 yields 

P 2 \Xi, Xi+ 1 , Xi+ 2 1 = P\ [Xi+I, Xi+ 2 1 = y i+ 2 


Finally, when x = x,-+i we have 


P\ [Xi, Xi+,1 = P [Xi+i, Xi+2] = yi+i 


so that 


P 2 [Xi, Xi+1, X i+2 \ 


(Xi +1 - Xi+ 2 )yi+i + (Xi 


Xi+i)yi+i 

-= y«+i 


Xi - Xi+2 
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Having established the pattern, we can now deduce the general recursive formula: 

PklXi, x i+] x i+k ] 

(X - Xi+k) Pk-l[XiX i+ i, Xi+k- ll + (Xi - X) Pk-llXi+iXi+ 2 , Xi+k ] 

= - (3.«J 

Xi Xi+k 

Given the value of x, the computations can be carried out in the following tabular 
format (shown for four data points): 


k = 0 k = l k = 2 k = 3 


Xo 

•P0U0] = yo 

P\[Xo,X 1] 

P 2 [Xq,X 1,X 2 \ 

P 3 [Xq, X\, x 2 , x 3 \ 

Xi 

II 

$ 

ce 

P\[X 1,X 2 ] 

P 2 [x L x 2 , X3I 


X 2 

Pq[x 2 ] = y 2 

P\[x 2 ,x 3] 



X 3 

P0IX3] = ys 





Table 3.2 


If the number of data points is m, the algorithm that computes the elements of 
the table is 

y = yData.copyO 
for k in range (l,m): 

for i in range(m-k): 

y[i] = ((x - xData[i+k])*y[i] + (xDatafi] - x)*y[i+l])/ \ 
(xData[i] - xData[i+k]) 

This algorithm works with the one-dimensional array y, which initially contains 
the y-values of the data (the second column in Table 3.2). Each pass through the outer 
loop computes the elements of y in the next column, which overwrite the previous 
entries. At the end of the procedure, y contains the diagonal terms of the table. The 
value of the interpolant (evaluated at x ) that passes through ali the data points is the 
first element of y. 

■ neville 

The following function implements Neville’s method; it returns P„[x) 

## module neville 

’’’ p = neville(xData,yData,x). 

Evaluates the polynomial interpolant p(x) that passes 
trough the specified data points by Neville’s method. 
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def neville(xData,yData,x): 

m = len(xData) # number of data points 
y = yData.copy() 
for k in range(l,m): 

y[0:m-k] = ((x - xData[k:m])*y[0:m-k] + \ 

(xData[0:m-k] - x)*y[1:m-k+1])/ \ 

(xData[0:m-k] - xData[k:m]) 

return y[0] 


Limitations of Polynomial Interpolation 

Polynomial interpolation should be carried out with the fewest feasible number of 
data points. Linear interpolation, using the nearest two points, is often sufficient 
if the data points are closely spaced. Three to six nearest-neighbor points produce 
good results in most cases. An interpolant intersecting more than six points must be 
viewed with suspicion. The reason is that the data points that are far from the point 
of interest do not contribute to the accuracy of the interpolant. In fact, they can be 
detrimental. 

The danger of using too many points is illustrated in Fig. 3.3. There are 11 equally 
spaced data points represented by the circles. The solid line is the interpolant, a poly¬ 
nomial of degree ten, that intersects all the points. As seen in the figure, a polynomial 
of such a high degree has a tendency to oscillate excessively between the data points. 
A much smoother resuit would be obtained by using a cubic interpolant spanning 
four nearest-neighbor points. 



x 

Figure 3.3. Polynomial interpolant displaying oscillations. 


Polynomial extrapolation (interpolating outside the range of data points) is dan- 
gerous. As an example, consider Fig. 3.4. There are six data points, shown as circles. 
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3.2 Polynomial Interpolation 


The fifth-degree interpolating polynomial is represented by the solid line. The inter¬ 
polant looks fine within the range of data points, but drastically departs from the 
obvious trend when x > 12. Extrapolating y at x = 14, for example, would be absurd 
in this case. 



Figure 3.4. Extrapolation may not follow the trend of data. 


If extrapolation cannot be avoided, the following two measures can be useful: 

• Plot the data and visually verify that the extrapolated value makes sense. 

• Use a low-order polynomial based on nearest-neighbor data points. A linear or 
quadratio interpolant, for example, would yield a reasonable estimate of y( 14) for 
the data in Fig. 3.4. 

• Work with a plot of log x vs. log y , which is usually much smoother than the x-y 
curve, and thus safer to extrapolate. Frequently this plot is almost a straight line. 
This is illustrated in Fig. 3.5, which represents the logarithmic plot of the data in 
Fig. 3.4. 



Figure 3.5. Logarithmic plot of the data in Fig. 3.4. 
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EXAMPLE 3.1 

Given the data points 


X 

0 

2 

3 

y 

7 

11 

28 


use Lagrange’s method to determine y at x = 1. 

Solution 

£ (x-x,)(x-x 2 ) (1 — 2) (1 — 3) 

0 Uo-x,)(xo-x 2 ) (0 — 2) (0 — 3) 

t = (x Xp) (x x 2 ) = (1 — 0)(1 — 3) 

1 (Xi - x 0 ) (Xi - x 2 ) (2 - 0) (2 - 3) 

= (x Xp) (x Xi) = (1 — 0)(1 — 2) 

2 (x 2 Xq) (x 2 Xi) (3 — 0)(3 - 2) 

7 28 

y = yo4 + yiG + #£> = - + n - y 

EXAMPLE 3.2 

The data points 


x 

-2 

1 

4 

-1 

3 

-4 

y 

-1 

2 

59 

4 

24 

-53 


lie on a polynomial. Determine the degree of this polynomial by constructing the 
divided difference table, similar to Table 3.1. 



Solution 


i 


y* 

vy,- 

v 2 y/ 

v 3 yi 

v 4 y ; - 

v 5 yi 

0 

-2 

-i 






1 

1 

2 

l 





2 

4 

59 

10 

3 




3 

-1 

4 

5 

-2 

1 



4 

3 

24 

5 

2 

1 

0 


5 

-4 

-53 

26 

-5 

1 

0 

0 


Here are a few sample calculations used in arriving at the figures in the table: 


Vy 2 

v 2 y 2 

V 3 y5 


y> - yo _ 59 - t-D 

x 2 - Xb 4 - (-2) 


Vy 2 — Vyi 10—1 


x 2 - Xi 


4-1 


= 3 


V 2 y 5 - v 2 y 2 - 5-3 


= 1 


x 5 -x 2 


-4-4 
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3.2 Polynomial Interpolation 


From the table we see that the last nonzero coefficient (last nonzero diagonal term) 
of Newton’s polynomial is V 3 y 3 , which is the coefficient of the cubic term. Hence the 
polynomial is a cubic. 

EXAMPLE 3.3 

Given the data points 


X 

4.0 

3.9 

3.8 

3.7 

y 

-0.06604 

-0.02724 

0.01282 

0.05383 


determine the root of y(x) = 0 by Neville’s method. 

Solution This is an example of inverse interpolation, where the roles of x and y are 
interchanged. Instead of computing y at a given x, we are finding x that corresponds 
to a given y (in this case, y = 0). Employing the format of Table 3.2 (with x and y 
interchanged, of course), we obtain 


i 

yt 

K 

II 

5 

PiU 

P 2 [, , 1 

hl,, ,1 

0 

-0.06604 

4.0 

3.8298 

3.8316 

3.8317 

1 

-0.02724 

3.9 

3.8320 

3.8318 


2 

0.01282 

3.8 

3.8313 



3 

0.05383 

3.7 





The following are sample computations used in the table: 

(y - yi) Polypi + (yo - y) Jhlyil 
y 0 - yi 

(0 + 0.02724) (4.0) + (-0.06604 - 0)(3.9) _ 

-0.06604 + 0.02724 

ty - yd Pi [yi, y>l + (yi - y) fi [y>, yd 

yi-ys 

(0 - 0.053831(3.8320) + (-0.02724 - 0) (3.8313) 

--—-----—-- = 3.8318 

-0.02724 - 0.05383 

All the P’s in the table are estimates of the root resulting from different orders 
of interpolation involving different data points. For example, P\ \ yo. yd is the root 
obtained from linear interpolation based on the first two points, and P> [ yi, y 2 , yd is 
the resuit from quadratic interpolation using the last three points. The root obtained 
from cubic interpolation over all four data points is x = P 3 [y 0 , yi, y2, y 3 ] = 3.8317. 


Pityo, yil = 


Pdyi, yz, yd = 
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EXAMPLE 3.4 

TtX 

The data points in the table lie on the plot of /(x) = 4.8 cos —. Interpolate this data by 
Newtoris method atx = 0,0.5,1.0,...,8.0 and compare the results with the “exact” 
values y t = /(x ; -). 


X 

0.15 

2.30 

3.15 

4.85 

6.25 

7.95 

y 

4.79867 

4.49013 

4.2243 

3.47313 

2.66674 

1.51909 


Solution 

#!/usr/bin/python 
## example3_4 

from numarray import array,arange 
from math import pi.cos 
from newtonPoly import * 

xData = arrayC[0.15,2.3,3.15,4.85,6.25,7.95]) 

yData = arrayC[4.79867,4.49013,4.2243,3.47313,2.66674,1.51909]) 

a = coeffts(xData,yData) 

print '’ x ylnterp yExact’’ 

print ' ’-’ ’ 

for x in arange(0.0,8.1,0.5) : 
y = evalPoly(a,xData,x) 
yExact = 4.8*cos(pi*x/20.0) 
print ’’%3.1f %9.5f %9.5f’’% (x,y,yExact) 
raw_input('’ \nPress return to exit’’) 

The results are: 


X 

ylnterp 

yExact 

1 

1 o 

■ 

1 o 

4.80003 

4.80000 

0.5 

4.78518 

4.78520 

1.0 

4.74088 

4.74090 

1.5 

4.66736 

4.66738 

2.0 

4.56507 

4.56507 

2.5 

4.43462 

4.43462 

3.0 

4.27683 

4.27683 

3.5 

4.09267 

4.09267 

4.0 

3.88327 

3.88328 
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4.5 

3.64994 

3.64995 

5.0 

3.39411 

3.39411 

5 . 5 

3.11735 

3.11735 

6.0 

2.82137 

2.82137 

6.5 

2.50799 

2.50799 

7.0 

2.17915 

2.17915 

7.5 

1.83687 

1.83688 

8.0 

1.48329 

1.48328 


3.3 Interpolation with Cubic Spline 


If there are more than a few data points, a cubic spline is hard to beat as a global 
interpolant. It is considerably “stiffer” than a polynomial in the sense that it has less 
tendency to oscillate between data points. 


Elastic strip 



Pins (data points) 
- x 


Figure 3.6. Mechanical model of natural cubic spline. 


The mechanical model of a cubic spline is shown in Fig. 3.6. It is a thin, elastic 
beam that is attached with pins to the data points. Because the beam is unloaded 
between the pins, each segment of the spline curve is a cubic polynomial—recall from 
beam theory that cfty/dx 4 = q/(EI), so that y(x) is a cubic since q = 0. At the pins, the 
slope and bending moment (and hence the second derivative) are continuous. There 
is no bending moment at the two end pins; consequently, the second derivative of 
the spline is zero at the end points. Since these end conditions occur naturally in the 
beam model, the resulting curve is known as the natural cubic spline. The pins, i.e., 
the data points, are called the knots of the spline. 



Figure 3.7 shows a cubic spline that spans n+ 1 knots. We use the notation 
fj + \ (x) for the cubic polynomial that spans the segment between knots i and i + 1. 
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Note that the spline is a piecewise cubic curve, put together from the n cubics 
/o i (x ), f 2 (x),..., f„-\jt(x), all ofwhich have different coefficients. 

Ifwe denote the second derivative ofthe spline atknot iby lq, continuityof second 
derivatives requires that 


fi-u(Xi) = fu+ M) = h 

At this stage, each k is unknown, except for 

ko = k n = 0 


(a) 


(3.9) 


The starting point for computing the coefficients of fj + 1 {x) is the expression for 
f'i' i+l {x), which we know to be linear. Using Lagrange’s two-point interpolation, we 
can write 


fi' i+ iW = kiliix ) + k i+ ili + i{x) 


where 


Therefore, 


(iW = 


- x i+1 
Xi - X i+ 1 


h+l{x) = 


X- Xj 
Xi+1 - Xi 


fu+ lU) = 


h{x - x i+ i) - k i+ 1 (x - x^ 


Xi - Xi+1 

Integrating twice with respect to x, we obtain 


fij+l (x) = 


kiix - x i+ 1 ) 3 - k i+ i(x - X;) 3 


+ A{x - Xi+i) - B{x - Xi] 


(b) 


(c) 


6 (x, - x i+ i) 

where A and B are constants of integration. The terms arising from the integration 
would usually be written as Cx + D. By letting C = A — B and D = — Ax ,+1 + Bxt, we 
end up with the last two terms of Eq. (c), which are more convenient to use in the 
computations that follow. 

Imposing the condition f ?i +i(Xi) = jy we get from Eq. (c) 

ki(_Xi - X;+i ) 3 


6 (x, - x i+ 1 ) 


+ A(Xi - Xi+i) = y t 


Therefore, 


A = 


-—-7T (Xj — X i+ i) 

Xi - Xi+1 6 


Similarly, f u +i{x i+ i) = y i+ 1 yields 


B=^± - ^±( Xi -Xi+i) 

Xi - Xi+1 6 


(d) 


(e) 
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Substituting Eqs. (d) and (e) into Eq. (c) results in 



(X-X/+i)(X/ -Xf+i) 


kj +1 r (x - x,-) 3 


- (X - x,- 


(x - X,-)(X/ - Xj+i) 


(3.10) 


6 lXi-Xi + i 


ytix-Xj+i)- y i+ i{x-Xj) 


Xi - X/+1 


The second derivatives fc,- of the spline at the interior knots are obtained from 
the slope continuity conditions f-_ 1 t [xi) = f ii+l (x,), where i = 1.2,...,«— l.Aftera 
little algebra, this results in the simultaneous equations 


Ar, : _i(x;_i - xi) + 2Ar, : (x i _i - x,-+i) + fc;+i(x ( - - x* +1 ) 


= 6 ("Jii—i= 1,2, 1 (3.11) 

\Xi_i-X/ Xi-X i+1 ) 

BecauseEqs. (3.11) have a tridiagonal coefficient matrix, they canbe solved econom- 
ically with the functions in module LUdecomp3 described in Section 2.4. 

If the data points are evenly spaced at intervals h, then x;_i — x; = x,- — x,-+i = — Ii, 
and the Eqs. (3.11) simplifyto 



(3.12) 


■ cubicSpline 

The first stage of cubic spline interpolation is to set up Eqs. (3.11) and solve them 
for the unknown k ’s (recall that k, = k n = 0). This task is carried out by the func- 
tion curvatures. The second stage is the computation of the interpolant at x from 
Eq. (3.10). This step can be repeated any number of times with different values 
of x using the function evalSpline. The function findSegment embedded in 
evalSpline finds the segment of the spline that contains x using the method of 
bisection. It returns the segment number; that is, the value of the subscript i in 
Eq. (3.10). 

## module cubicSpline 

’’’ k = curvatures(xData,yData). 

Returns the curvatures {k}of cubic spline at the knots. 


y = evalSpline(xData,yData,k,x). 

Evaluates cubic spline at x. The curvatures {k} can be 
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computed with the function ’curvatures’. 

from numarray import zeros,ones,Float64,array 
from LUdecomp3 import * 

def curvatures(xData,yData): 
n = len(xData) - 1 
c = zeros((n),type=Float64) 
d = ones((n+1),type=Float64) 
e = zeros((n),type=Float64) 
k = zeros((n+1),type=Float64) 
c[0:n-l] = xData[0:n-l] - xData[l:n] 
d[l:n] = 2.0*(xData[0:n-1] - xData[2:n+1]) 
e[l:n] = xData[l:n] - xData[2:n+l] 
k[l:n] =6.0*(yData[0:n-1] - yData[l:n]) \ 

/(xData[0:n-1] - xData[l:n]) \ 

-6.0*(yData[1:n] - yData[2:n+1]) \ 

/(xData[l:n] - xData[2:n+1]) 

LUdecomp3(c,d,e) 

LUsolve3(c,d,e,k) 
return k 

def evalSpline(xData,yData,k,x): 

def findSegment(xData,x): 
iLeft = 0 

iRight = len(xData)- 1 
while 1: 

if (iRight-iLeft) <= 1: return iLeft 

i =(iLeft + iRight)/2 

if x < xData[i]: iRight = i 

else: iLeft = i 

i = findSegment(xData,x) # Find the segraent spanning x 

h = xData[i] - xData[i+l] 

y = ((x - xData[i+l])* *3/h - (x - xData[i+l])*h)*k[i]/6.0 \ 
- ((x - xData[i])* *3/h - (x - xData[i])*h)*k[i+l]/6.0 \ 

+ (yData[i]*(x - xData[i+l]) \ 

- yData[i+1]*(x - xData[i]))/h 


return y 
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EXAMPLE 3.5 

Use natural cubic spline to determine y at x = 1.5. The data points are 


X 

i 

2 

3 

4 

5 

y 

0 

1 

0 

1 

0 


Solution The five knots are equally spaced at h = 1. Recalling that the second deriva¬ 
tive of a natural spline is zero at the first and last knot, we have = /c, = 0. The second 
derivatives at the other knots are obtained from Eq. (3.12). Using i = 1, 2, 3 results in 
the simultaneous equations 

0 + 4fci + k 2 = 6 [0 — 2(1) + 0] = -12 

k\ + 4k 2 + ka = 6 [1 - 2(0) + 1] = 12 

k 2 + 4k 3 + 0 = 6 [0 - 2(1) + 0] = -12 

The solution is fci = k 3 = —30/7, k 2 = 36/7. 

The point x = 1.5 lies in the segment between knots 0 and 1. The corresponding 
interpolantis obtained fromEq. (3.10) bysetting i = 0. With x, — jq + i = — h= — 1, we 
obtain from Eq. (3.10) 

/o, i (x) = - g’ [(x- Xif - (x- Xi)] + [(x- x 0 ) 3 - (x - Xo)] 

- [yo(x — Xi) - yi(x-xb)] 


Therefore, 


y(1.5) = /o,i(1.5) 

= ° + J (-y") [(1.5 - l) 3 - (1.5 - 1)] - [0 - 1(1.5 - 1)] 
= 0.7679 


The plot of the interpolant, which in this case is made up of four cubic segments, is 
shown in the figure. 



X 
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EXAMPLE 3.6 

Sometimes it is preferable to replace one or both of the end conditions of the cu- 
bic spline with something other than the natural conditions. Use the end condition 
fo jfO) =0 (zeroslope),ratherthan f ", (0) = 0 (zerocurvature), todeterminethecubic 
spline interpolant at x = 2.6, given the data points 


X 

0 

1 

2 

3 

y 

1 

1 

0.5 

0 


Sol ution We must first modify Eqs. (3.12) to account for the new end condition. Setting 
i = 0 in Eq. (3.10) and differentiating, we get 


/o.iM = j 


(x - Xi) 2 ' 

3-(xb - xi) 


Xo-Xi 

Thus the end condition fg 1 (jtp) = 0 yields 


h 

6 


(X - Xo) 2 . 

3-(Xo - Xi) 

Xo - x. 


+ 


yp- u 

Xo - Xi 


^ (Xo - Xi) + ^ [Xo - Xi) + ——— = 0 
3 6 Xo - Xi 


or 


2/d{) -f- 1c\ = —6 


yo-yi 


(x 0 - Xi ) 2 

From the given data we see that y 0 = y t = 1, so that the last equation becomes 

2 l<o T k\ = 0 

The other equations in Eq. (3.12) are unchanged. Knowing that fe, = 0, we have 
ko + 4fci + k 2 = 6 [1 - 2(1) + 0.5] = -3 
fci + 4fc 2 = 6 [1 - 2(0.5) + 0] = 0 


(a) 

(b) 

(c) 


The solution of Eqs. (a)-(c) is ko = 0.4615, fci = —0.9231, k 2 = 0.2308. 

The interpolant can now be evaluated from Eq. (3.10). Substituting i = 2 and 
Xi — Xi +1 = — 1, we obtain 

/ 2 , 3 M = ^ [-(x - x 3 ) 3 + (x - x 3) ] - ^ [-(x - x 2 ) 3 + (x - x 2 )] 

- y 2 [x - x 3 ) + y?(x - x 2 ) 


Therefore, 


0.2308 r , 

y(2.6) = / 2 , 3 (2.6) = —— [—(—0.4) 3 + (-0.4)] - 0 - 0.5(-0.4) + 0 


= 0.1871 


6 
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EXAMPLE 3.7 

Utilize the module cubicSpline to write a program that interpolates between given 
data points with natural cubic spline. The program must be able to evaluate the 
interpolant for more than one value of x. As a test, use data points specified in Example 
3.4 and compute the interpolant at x = 1.5 and x = 4.5 (duetosymmetry, thesevalues 
shouldbe equal). 

Solution 

#!/usr/bin/python 
## example3_7 

from numarray import array,Float64 
from cubicSpline import * 

xData = arrayC[1,2,3,4,5],type=Float64) 
yData = array([0,1,0,1,0],type=Float64) 
k = curvature s(xData,yData) 
while 1: 

try: x = eval(raw_input('’\nx ==> ’’)) 
except SyntaxError: break 

print ’’y evalSpline(xData,yData,k,x) 

raw_input('’Done. Press return to exit’’) 

Running the program produces the following resuit: 

x ==> 1.5 

y = 0.767857142857 

x ==> 4.5 

y = 0.767857142857 

x ==> 

Done. Press return to exit 


PROBLEM SET3.1 

1. Given the data points 


X 

-1.2 

0.3 

1.1 

y 

-5.76 

-5.61 

-3.69 


determine y at x = 0 using (a) Neville's method; and (b) Lagrange’s method. 
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2. Find the zero of y[x) from the following data: 


X 

0 

0.5 

1 

1.5 

2 

2.5 

3 

y 

1.8421 

2.4694 

2.4921 

1.9047 

0.8509 

-0.4112 

-1.5727 


Use Lagrange’s interpolation over (a) three; and (b) four nearest-neighbor data 
points. Hint: after finishing part (a), part (b) can be computed with a relatively 
small effort. 

3. The function y(x) represented by the data in Prob. 2 has a maximum at x = 0.7679. 
Compute this maximum by Neville’s interpolation over four nearest-neighbor 
data points. 

4. Use Neville’s method to compute y at x = jt /4 from the data points 


X 

0 

0.5 

1 

1.5 

2 

y 

-1.00 

1.75 

4.00 

5.75 

7.00 


5. Given the data 


X 

0 

0.5 

1 

1.5 

2 

y 

-0.7854 

0.6529 

1.7390 

2.2071 

1.9425 


find y at x = jt/4 and at jt/ 2. Use the method that you consider to be most 
convenient. 

6 . The points 


X 

-2 

1 

4 

-1 

3 

-4 

y 

-1 

2 

59 

4 

24 

-53 


lie on a polynomial. Use the divided difference table of Newton’s method to de- 
termine the degree of the polynomial. 

7. Use Newton’s method to find the polynomial that fits the following points: 


X 

-3 

2 

-1 

3 

1 

y 

0 

5 

-4 

12 

0 


8 . Use Neville’s method to determine the equation of the quadratic that passes 
through the points 


X 

-i 

1 

3 

y 

17 

-7 

-15 
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3.3 Interpolation with Cubic Spline 


9. The density of air p varies with elevation h in the following manner: 


h (km) 

0 

3 

6 

p (kg/m 3 ) 

1.225 

0.905 

0.652 


Express p ( h) as a quadratic function using Lagrange’s method. 

10. Determine the natural cubic spline that passes through the data points 


X 

0 

i 

2 

y 

0 

2 

1 


Note that the interpolant consists of two cubics, one valid in 0 < x < 1, the other 
in 1 < x < 2. Verify that these cubics have the same first and second derivatives 
at x = 1. 

11. Given the data points 


X 

i 

2 

3 

4 

5 

y 

13 

15 

12 

9 

13 


determine the natural cubic spline interpolant at x = 3.4. 

12. Compute the zero of the function y(x) from the following data: 


X 

0.2 

0.4 

0.6 

0.8 

1.0 

y 

1.150 

0.855 

0.377 

-0.266 

-1.049 


Use inverse interpolation with the natural cubic spline. Hint: reorder the data so 
that the values of y are in ascending order. 

13. Solve Example 3.6 with a cubic spline that has constant second derivatives within 
its first and last segments (the end segments are parabolic). The end conditions 
for this spline are ko = k\ and k n - i = k n . 

14. ■ Write a computer program for interpolation by Neville’s method. The program 
must be able to compute the interpolant at several user-specified values of x. 
Test the program by determining y at x = 1.1, 1.2 and 1.3 from the following 
data: 


X 

-2.0 

-0.1 

-1.5 

0.5 

y 

2.2796 

1.0025 

1.6467 

1.0635 

X 

-0.6 

2.2 

1.0 

1.8 

y 

1.0920 

2.6291 

1.2661 

1.9896 


(Answer: y = 1.3262, 1.3938, 1.4693) 
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15. ■ The specific heat c p of aluminum depends on temperature T as follows: 6 


T (°C) 

-250 

-200 

-100 

0 

100 

300 

c p (kJ/kg-K) 

0.0163 

0.318 

0.699 

0.870 

0.941 

1.04 


Determine c p at T = 200°C and 400°C. 

16. ■ Find y at x = 0.46 fromthe data 


X 

0 

0.0204 

0.1055 

0.241 

0.582 

0.712 

0.981 

y 

0.385 

1.04 

1.79 

2.63 

4.39 

4.99 

5.27 


17. ■ The table shows the drag coefficient c D of a sphere as a function of Reynolds 
number Re. 7 Use natural cubic spline to find c D at Re = 5,50,500 and 5000. Hini: 
use log-log scale. 


Re 

0.2 

2 

20 

200 

2000 

20 000 

Cd 

103 

13.9 

2.72 

0.800 

0.401 

0.433 


18. ■ Solve Prob. 17 using a polynomial interpolant intersecting four nearest- 
neighbor data points. 

19. ■ The kinematic viscosity /x k of water varies with temperature T in the following 
manner: 


T (°C) 

0 

21.1 

37.8 

54.4 

71.1 

87.8 

100 

/x k (10“ 3 m 2 /s) 

1.79 

1.13 

0.696 

0.519 

0.338 

0.321 

0.296 


Interpolate /x k at T = 10°, 30°, 60° and 90°C. 

20. ■ The table shows how the relative density p of air varies with altitude h. Determine 
the relative density of air at 10.5 km. 


h (km) 

0 

1.525 

3.050 

4.575 

6.10 

7.625 

9.150 

P 

1 

0.8617 

0.7385 

0.6292 

0.5328 

0.4481 

0.3741 


3.4 Least-Squares Fit 
OverView 

If the data are obtained from experiments, they typically contain a significant amount 
of random noise due to measurement errors. The task of curve fitting is to find a 

6 Source: Black, Z. B., and Hartley, J. G., Thermodynamics, Harper & Row, 1985. 

7 Source: Kreith, F., Principies ofHeat Transfer, Harper & Row, 1973. 
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3.4 Least-Squares Fit 


smooth curve that fits the data points “on the average.” This curve should have a 
simple form (e.g., a low-order polynomial), so as to not reproduce the noise. 

Let 


/(x) = f{x;ao,cii,...,a n7 ) 

be the function that is to be fitted to the n+ 1 data points (x y{), i = 0,1,..., n. The 
notation implies that we have a function of x that contains m+ 1 variable parameters 
«o . «i,.... a m , where m< n. The form of /(x) is determined beforehand, usually from 
the theory associated with the experiment from which the data is obtained. The only 
means of adjusting the fit is the parameters. For example, if the data represent the 
displacements y,- of an overdamped mass-spring system at time tj, the theory suggests 
the choice f{t) = aote~ ait . Thus curve fitting consists of two steps: choosing the form 
of /(x), followed by computation of the parameters that produce the best fit to the 
data. 

This brings us to the question: what is meant by “best” fit? ff the noise is confined 
to the y-coordinate, the most commonly used measure is the least-squaresfit, which 
minimizes the function 

n 

S(«b, = [y; - /M] 2 (3.13) 

!=0 

with respect to each ci/. Therefore, the optimal values of the parameters are given by 
the solution of the equations 

d S 

— = 0, k=0, 1, ...,m (3.14) 

The terms r,- = y, — /(x,) in Eq. (3.13) are called residuals; they represent the discrep- 
ancy between the data points and the fitting function at x,-. The function S to be 
minimized is thus the sum of the squares of the residuals. Equations (3.14) are gen- 
erally nonlinear in aj and may thus be difficult to solve. Often the fitting function is 
chosen as a linear combination of specified functions fj (x): 

/(x) = an / 0 (x) + «i /i(x) H-b a m f„{x ) 

in which case Eqs. (3.14) are linear. If the fitting function is a polynomial, we have 
/o(x) = 1, /i(x) = x, / 2 (x) = x 2 , etc. 

The spread of the data about the fitting curve is quantified by the Standard devi- 
ation, defined as 



Note that if n= f/? we have interpolation, not curve fitting. In that case both the 
numerator and the denominator in Eq. (3.15) are zero, so that a is indeterminate. 
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Fitting a Straight Line 

Fitting a straight line 


/(x) = a+bx (3.16) 

to data is also known as linear regression. In this case the function to be minimized is 

n n 

S{a, b) = [y,- - /te-)] 2 = J2(yi~a- bxj) 2 


i =0 


i=0 


Equations (3.14) nowbecome 

3 S ^ 

— = Y — 2(y f - a - bxi ) = 2 


dS 


i =0 


i{n+ 1) + b^xt ~Yy t 


i =0 


= 0 


1=0 


— = -2(yi - a - bXi)Xi = 2 ( a Y x i + b Y x f ~ Y x «') = 0 


i =0 


Dividing both equations by 2 (n+ 1) and rearranging terms, we get 

a+ * b = f ia+ (7^- l f s 2 b= ^ii™ 


where 


x = 


i n 

— Y- 

n+ 1 


y = 


n + 


tI> 


/=o i=0 

are the mean values of the x and y data. The solution for the parameters is 

a= yJ2 x t -xExiyi b= E*;y;-*Ey* 


(3.17) 


(3.18) 


E x? - (n+ l)x 2 E x f - ( w + l)* 2 

These expressions are susceptible to roundoff errors (the two terms in each numerator 
as well as in each denominator can be roughly equal). It is better to compute the 
parameters from 


Ey;(JQ -x) _ 

b = =—t-- a =y -xb 

E XiiXi -x) 

which are equivalent to Eqs. (3.18), but much less affected by rounding off. 


(3.19) 


Fitting Linear Forms 

Consider the least-squares fit of the linear form 

m 

/(x) = Oofo (x) + a 1 fi (x) H-f a m f m {x) = Y a, fj (x) 

i=o 


(3.20) 
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3.4 Least-Squares Fit 


whereeach fj (x) isapredeterminedfunctionofx, called a basis function. Substitution 
in Eq. (3.13) yields 

„ -|2 


*=E 


1=0 


yi E r '/ 

j =o 


Thus Eqs. (3.14) are 


— =-2| e 

d ajc I ^ 


i=0 


7=0 


fk(Xi)\ = 0, k=0, l,...,m 


Dropping the constant (—2) and interchanging the order of summation, we get 


E 


E fjwfk&i' 


y=o L *=o 


«j = E f^ x i)yi> k = o,i. 


?n 


i=0 


In matrix notation these equations are 


Aa = b 


(3.21a) 


where 

n n 

A v = E f^ Xi) ht = E M x ^y (3.2 ib) 

/=0 i=0 

Equations (3.21a), known as the normal equations of the least-squares fit, can be 
solved with the methods discussed in Chapter 2. Note that the coefficient matrix is 
symmetric, i.e., Ay = A^. 


Polynomial Fit 

A commonly used linear form is a polynomial. If the degree of the polynomial is m, 
we have /(x) = EJLo a jX k - Here the basis functions are 

fjix) = x J (7 = 0,1. m) (3.22) 

so that Eqs. (3.21b) become 

A kj = E x i +k ^=E 



n 

£*; 

Exf ■ 

■ Exr ' 


'Eyt 

A = 

E x i 

Exf 

E*? ■ 

• Exr 1 

b = 

E xiyi 


.Exr 1 

E*r 

Exr 1 . 

■ Exf m _ 


_ E^ vi. 


(3.23) 
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where stands for J2l=o -The normal equations become progressively ill-conditioned 
with increasing m Fortunately, this is of little practical consequence, because only 
low-order polynomials are useful in curve fitting. Polynomials of high order are 
not recommended, because they tend to reproduce the noise inherent in the 
data. 

■ polyFit 

The function polyFit in this module sets up and solves the normal equations for 
the coefficients of a polynomial of degree m It returns the coefficients of the poly- 
nomial. To facilitate computations, the terms n, J2 x h x f m that make up 

the coefficient matrix in Eq. (3.23) are first stored in the vector s and then inserted 
into A. The normal equations are then solved by Gauss elimination with pivoting. 
Following the solution, the Standard deviation n can be computed with the func¬ 
tion stdDev. The polynomial evaluation in stdDev is carried out by the embedded 
function evalPoly—see Section 4.7 for an explanation of the algorithm. 

## module polyFit 

’’’ c = polyFit(xData,yData,m). 

Returns coefficients of the polynomial 
p(x) = c[0] + c[l]x + c[2]x~2 +...+ c[m]x~m 
that fits the specified data in the least 
squares sense. 

sigma = stdDev(c,xData,yData) . 

Computes the std. deviation between p(x) 
and the data. 

from numarray import zeros,Float64 
from math import sqrt 
from gaussPivot import * 

def polyFit(xData,yData,m): 

a = zeros((m+1,m+1),type=Float64) 
b = zeros((m+1),type=Float64) 
s = zeros((2*m+l),type=Float64) 
for i in range(len(xData)): 
temp = yData[i] 
for j in range(m+1): 

b[j] = b [j ] + temp 
temp = temp*xData[i] 





129 


3.4 Least-Squares Fit 


temp = 1.0 

for j in range(2*m+l): 
s[j] = s[j] + temp 
temp = temp*xData[i] 
for i in range(m+l): 

for j in range(m+l): 
a[i,j] = s[i+j] 
return gaussPivot(a,b) 

def stdDev(c,xData,yData): 

def evalPoly(c,x): 
m = len(c) - 1 
p = c[m] 

for j in range(m): 


p = p*x + c[m-j-1] 


return p 

n = len(xData) - 1 
m = len(c) - 1 
sigma = 0.0 
for i in range(n+l): 

p = evalPolyCc,xData[i]) 
sigma = sigma + CyData[i] - p)**2 
sigma = sqrt(sigma/(n - m)) 
return sigma 

Weighting of Data 

There are occasions when our confidence in the accuracy of data varies from point to 
point. For example, the instrument taking the measurements may be more sensitive 
in a certain range of data. Sometimes the data represent the results of several exper- 
iments, each carried out under different conditions. Under these circumstances we 
may want to assign a confidence factor, or weight, to each data point and minimize 
the sum of the squares of the weighted residuals ri = W, [y,- — /(x,)], where IV/ are the 
weights. ffence the function to be minimized is 


n 



(3.24) 


This procedure forces the fitting function /(x) closer to the data points that have 
higher weights. 
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Weighted linear regression 

If the fitting function is the straight line /(x) = a+ bx, Eq. (3.24) becomes 

n 

S(a, b) = Y^ w f ( Vi ~a~ bxi ) 2 (3.25) 

i =0 

The conditions for minimizing S are 

= -2 Wfiyt -a- bxO = 0 


3 S ^ 

—r = -2 ^ Wf [y t - a - bx^Xi = 0 


aJ^Wf + bJ^ Wf Xi = J2 Wfy , (3.26a) 

i= 0 i= 0 i= 0 

aJ2 W i X i + b it< W i^ = IZ W hiyi 

i =0 i=0 i =0 

Dividing Eq. (3.26a) by X! Wf and introducing the weighted averages 

E wfxi Ewfyi 
E wf E wf 

we obtain 

a = j> — fo£ 

Substituting into Eq. (3.26b) and solving for foyields after some algebra 

b= E WfyitXi ~ x) 

E WfxdXi - x) 

Note that Eqs. (3.28) are quite similar to Eqs. (3.19) for unweighted data 


(3.26b) 

(3.27) 

(3.28a) 

(3.28b) 


Fitting exponential functions 

A special application of weighted linear regression arises in fitting various exponential 
functions to data. Consider as an example the fitting function 

/(x) = ae bx 

Normally, the least-squares fit would lead to equations that are nonlinear in a and b. 
But if we fit In y rather than y, the problem is transformed to linear regression: fit the 
function 


F (x) = ln /(x) = ln a + bx 
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to the data points (x,-, ln y), / = 0,1....,«. This simplification comes at a price: least- 
squares fit to the logarithm of the data is not quite the same as the least-squares fit to 
the original data. The residuals of the logarithmic fit are 

Ri = ln y - F (x,0 = ln y - (ln a + bx t ) (3.29a) 

whereas the residuals used in fitting the original data are 

n = y t - f(Xi) = y t - ae hXi (3.29b) 

This discrepancy can be largely eliminated by weighting the logarithmic fit. From 
Eq. (3.29b) we obtain ln(r,- — y ) = \n{ae hx ‘) = ln a+ bx,, so Lhat Hq. (3.29a) canbewrit- 
ten as 

Ri = ln y - ln(r, - y) = ln ^1 - y J 

ff the residuals r, are sufficiently small (r,- << y), we can use the approximation 
ln(l - Tityi) « Tityi, so that 


Ri ~ Tityi 

We can now see that by minimizing Yt, Rf > we have inadvertently introduced the 
weights 1/y. This effect can be negated if we apply the weights W, = y when fitting 
F(x) to (lny, xt). That is, minimizing 

n 

s=J2tf R i (3 - 30) 

i =0 

is a good approximation to minimizing Y r f- 

Other examples that also benefit from the weights VV, = y are given in Table 3.3. 


/M 

Fix) 

Data to be fitted by F (x) 

axe 

ln [ /(x) /x] = In a + bx 

[xt, lnfy/xd 

ax b 

ln / (x) = ln a+ Mn(x) 

(lnxi.lny) 


Table 3.3 


EXAMPLE 3.8 

Fit a straight line to the data shown and compute the Standard deviation. 


X 

0.0 

1.0 

2.0 

2.5 

3.0 

y 

2.9 

3.7 

4.1 

4.4 

5.0 
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Solution The averages of the data are 

1 0.0+1.0+ 2.0+ 2.5+ 3.0 

X '=^L*‘' =- 


5^ 5 

2.9+ 3.7+ 4.1 + 4.4+ 5.0 


= 1.7 


y=\T,y» = 


= 4. 02 


5 5 

The intercept a and slope b of the interpolant can now be determined from Eq. (3.19): 

Ey;ta -x) 


b = 


J2 x d x i -x) 

2.9(—1.7) + 3.7(—0.7) + 4.1(0.3) + 4.4(0.8) + 5.0(1.3) 
0.0(—1.7) + 1.0(—0.7) + 2.0(0.3) + 2.5(0.8) + 3.0(1.3) 
3.73 

= 0.6431 


5.8 


a=y-xb= 4.02 - 1.7(0.6431) = 2.927 

Therefore, the regression line is /(x) = 2.927 + 0.6431x, which is shown in the figure 
together with the data points. 



We start the evaluation of the Standard deviation by computing the residuals: 


X 

0.000 

1.000 

2.000 

2.500 

3.000 

y 

2.900 

3.700 

4.100 

4.400 

5.000 

/w 

2.927 

3.570 

4.213 

4.535 

4.856 

y - /M 

-0.027 

0.130 

-0.113 

-0.135 

0.144 
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The sum of the squares of the residuals is 

S = J2 [ y* - /wi 2 

= (-0.027) 2 + (0.130) 2 + (-0.113) 2 + (-0.135) 2 + (0.144) 2 = 0.06936 


so that the Standard deviation in Eq. (3.15) becomes 


a = 



0.06936 

5-2 


0.1520 


EXAMPLE 3.9 

Determine the parameters a and b so that /(x) = ae bx fits the following data in the 
least-squares sense. 


X 

1.2 

2.8 

4.3 

5.4 

6.8 

7.9 

y 

7.5 

16.1 

38.9 

67.0 

146.6 

266.2 


Use two different methods: (1) fit in y,; and (2) fit ln y, with weights Wi = y,-. Compute 
the Standard deviation in each case. 

Solution of Part (1) The problem is to fit the function ln(ne fa ) = ln a + bx to the data 


X 

1.2 

2.8 

4.3 

5.4 

6.8 

7.9 

II 

2.015 

2.779 

3.661 

4.205 

4.988 

5.584 


We are now dealing with linear regression, where the parameters to be found are 
A = ln a and b. Following the steps in Example 3.8, we get (skipping some of the 
arithmetic details) 


x = 



4.733 



3.872 


b _ J^ZiiXj-X) 

Y.xAxi-x) 


16.716 

31.153 


0. 5366 


A= z — xb = 1.3323 


Therefore, a = e 21 = 3.790 and the fitting function becomes /(x) = 3.790e° 5366 . The 
plots of /(x) and the data points are shown in the figure. 
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Here is the computation of Standard deviation: 


X 

1.20 

2.80 

4.30 

5.40 

6.80 

7.90 

y 

7.50 

16.10 

38.90 

67.00 

146.60 

266.20 

f(x) 

7.21 

17.02 

38.07 

68.69 

145.60 

262.72 

y- m 

0.29 

-0.92 

0.83 

-1.69 

1.00 

3.48 


s = J2iy *- /mi 2 = 17 -59 


a = 


6-2 


2.10 


As pointed out before, this is an approximate solution of the stated problem, since 
we did not fit y,-, but ln y,-. Judging by the plot, the fit seems to be quite good. 


Solution of Part (2) We again fit \n(ae hx ) = 1 n a + bx lo z= In y. but this time the 
weights Wi = yt are used. From Eqs. (3.27) the weighted averages of the data are (recall 
that we fit z = ln y) 


£ E Jf *■ 
T.yfzi 


737.5 x 10 3 
98.67 x 10 3 


528.2 x 10 3 

-- = 5.353 

98.67 x 10 3 


and Eqs. (3.28) yield for the parameters 

, E yf z d x i — x) 35.39 x 10 3 

b= -- =-^ = 0.5440 

E yfjqUi - i) 65.05 x 10 3 


ln a = z - bx = 5.353 - 0.5440(7.474) = 1.287 
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Therefore, 

a=^ na = e 1287 = 3.622 

so that the fitting function is f{x) = 3.622e° 544te . As expected, this resuit is somewhat 
different from that obtained in Part (1). 

The computations of the residuals and the Standard deviation are as follows: 


X 

1.20 

2.80 

4.30 

5.40 

6.80 

7.90 

y 

7.50 

16.10 

38.90 

67.00 

146.60 

266.20 

/M 

6.96 

16.61 

37.56 

68.33 

146.33 

266.20 

y- m 

0.54 

-0.51 

1.34 

-1.33 

0.267 

0.00 


/Mi' =4-186 


a = J S = 1.023 
V 6-2 

Observe that the residuals and Standard deviation are smaller than in Part (1), indi- 
cating a better fit, as expected. 

It can be shown that fitting y,- directly (which involves the solution of a transcen- 
dental equation) resultsin f(x) = 3.614e 0 5442 r . The corresponding Standard deviation 
is a = 1.022, which is very close to the resuit in Part (2). 

EXAMPLE 3.10 

Write a program that fits a polynomial of arbitrary degree m to the data points shown 
below. Use the program to determine m that best hts this data in the least-squares 
sense. 


X 

-0.04 

0.93 

1.95 

2.90 

3.83 

5.00 

y 

-8.66 

-6.44 

-4.36 

-3.27 

-0.88 

0.87 

X 

5.98 

7.05 

8.21 

9.08 

10.09 


y 

3.31 

4.63 

6.19 

7.40 

8.85 



Solution The program shown below prompts for m Execution is terminated by en- 
tering an invalid character (e.g., the “return” character). 

#!/usr/bin/python 
## example3_10 
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from numarray import array 
from polyEit import * 

xData = arrayC[-0.04,0.93,1.95,2.90,3.83,5.0, \ 

5.98,7.05,8.21,9.08,10.09]) 
yData = arrayC[-8.66,-6.44,-4.36,-3.27,-0.88,0.87, \ 
3.31,4.63,6.19,7.4,8.85]) 

while 1: 
try: 

m = eval(raw_input(’’ \nDegree of polynomial ==> ’’)) 
coeff = polyFit(xData,yData,m) 
print ’’Coefficients are:\n’’,coeff 

print ’’Std. deviation =’’,stdDev(coeff,xData,yData) 
except SyntaxError: break 
raw_input(’’Finished. Press return to exit’’) 

The results are: 

Degree of polynomial ==> 1 
Coefficients are: 

[-7.94533287 1.72860425] 

Std. deviation = 0.511278836737 

Degree of polynomial ==> 2 
Coefficients are: 

[-8.57005662 2.15121691 -0.04197119] 

Std. deviation = 0.310992072855 

Degree of polynomial ==> 3 
Coefficients are: 

[-8,46603423e+00 1.98104441e+00 2.88447008e-03 -2.98524686e-03] 

Std. deviation = 0.319481791568 

Degree of polynomial ==> 4 
Coefficients are: 

[ -8.45673473e+00 1.94596071e+00 2.06138060e-02 

-5.82026909e-03 1.41151619e-04] 

Std. deviation = 0.344858410479 

Degree of polynomial ==> 

Finished. Press return to exit 




137 


3.4 Least-Squares Fit 


Because the quadratic f[x) = —8.5700 + 2.1512x — 0.041971x 2 produces the 
smallest Standard deviation, it can be considered as the “best” fit to the data. But 
be warned—the Standard deviation is not a reliable measure of the goodness-of-fit. 
It is always a good idea to plot the data points and /(x) before final determination 
is made. The plot of our data indicates that the quadratic (solid line) is indeed a 
reasonable choice for the fitting function. 



PROBLEM SET3.2 

Instructioris Plot the data points and the fitting function whenever appropriate. 

1. Show that the straight line obtained by least- squares fit of unweighted data always 
passes through the point (x, y). 

2. Use linear regression to find the line that fits the data 


X 

-1.0 

-0.5 

0 

0.5 

1.0 

y 

-1.00 

-0.55 

0.00 

0.45 

1.00 


and determine the Standard deviation. 

3. Three tensile tests were carried out on an aluminum bar. In each test the strain 
was measured at the same values of stress. The results were 


Stress (MPa) 

34.5 

69.0 

103.5 

138.0 

Strain (Test 1) 

0.46 

0.95 

1.48 

1.93 

Strain (Test 2) 

0.34 

1.02 

1.51 

2.09 

Strain (Test 3) 

0.73 

1.10 

1.62 

2.12 


where the units of strain are mm/ m. Use linear regression to estimate the modulus 
of elasticity of the bar (modulus of elasticity = stress/strain). 

4. Solve Prob. 3 assuming that the third test was performed on an inferior machine, 
so that its results carry only half the weight of the other two tests. 
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5. ■ Fit a straight line to the following data and compute the Standard deviation. 


X 

0 

0.5 

1 

1.5 

2 

2.5 

y 

3.076 

2.810 

2.588 

2.297 

1.981 

1.912 

X 

3 

3.5 

4 

4.5 

5 


y 

1.653 

1.478 

1.399 

1.018 

0.794 



6 . ■ Thetabledisplaysthemass Mandaveragefuelconsumptioni ofmotorvehicles 
manufactured by Ford and Honda in 1999. Fit a straight line <f> = a+ bM to the 
data and compute the Standard deviation. 


Model 

M(kg) 

i (km/liter) 

Contour 

1310 

10.2 

Crown Victoria 

1810 

8.1 

Escort 

1175 

11.9 

Expedition 

2360 

5.5 

Explorer 

1960 

6.8 

F-150 

2020 

6.8 

Ranger 

1755 

7.7 

Taurus 

1595 

8.9 

Accord 

1470 

9.8 

CR-V 

1430 

10.2 

Civic 

1110 

13.2 

Passport 

1785 

7.7 


7. ■ The relative density p of air was measured at various altitudes h. The results 
were: 


h (km) 

0 

1.525 

3.050 

4.575 

6.10 

7.625 

9.150 

P 

1 

0.8617 

0.7385 

0.6292 

0.5328 

0.4481 

0.3741 


Use a quadratic least-squares fit to determine the relative air density at h = 
10.5 km. (This problem was solved by interpolation in Prob. 20, Problem 
Set3.1.) 

8 . ■ The kinematic viscosity /_t k of water varies with temperature T as shown in the 
table. Determine the cubic that best fits the data, and use it to compute /x k at 
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3.4 Least-Squares Fit 


T = 10°, 30°, 60° and90°C. (This problem was solved in Prob. 19, ProblemSet3.1 
by interpolation.) 


7TC) 

0 

21.1 

37.8 

54.4 

71.1 

87.8 

100 

/X k (10“ 3 m 2 /s) 

1.79 

1.13 

0.696 

0.519 

0.338 

0.321 

0.296 


9. ■ Fit a straight line and a quadratic to the data 


X 

1.0 

2.5 

3.5 

4.0 

1.1 

1.8 

2.2 

3.7 

y 

6.008 

15.722 

27.130 

33.772 

5.257 

9.549 

11.098 

28.828 


Which is a better fit? 

10. ■ The table displays thermal efficiencies ofsome earlysteam engines. 8 Determine 
the polynomial that provides the best fit to the data and use it to predict the thermal 
efficiency in the year 2000. 


Year 

Efficiency (%) 

Type 

1718 

0.5 

Newcomen 

1767 

0.8 

Smeaton 

1774 

1.4 

Smeaton 

1775 

2.7 

Watt 

1792 

4.5 

Watt 

1816 

7.5 

Woolf compound 

1828 

12.0 

Improved Cornish 

1834 

17.0 

Improved Cornish 

1878 

17.2 

Corliss compound 

1906 

23.0 

Triple expansion 


11. The table shows the variation of the relative thermal conductivity k of sodium with 
temperature T. Find the quadratic that fits the data in the least-squares sense. 


T (°Q 

79 

190 

357 

524 

690 

k 

1.00 

0.932 

0.839 

0.759 

0.693 


12. Let /(x) = ax b be the least-squares fit of the data (jc/, y), i = 0, 1, n, and let 
F(x) = In a + Mn jebe the least-squares fit offlnx/, lny )—see Table 3.3. Prove that 


8 Source: Singer, C., Holmyard, E. 1., Hali, A. R., and Williams, T. H., A History of Technology, Oxford 
University Press, 1958. 
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Interpolation and Curve Fitting 


Rj ss ri/yi, where the residuals are r,- = y, — f{x{) and i?,- = 1 n y,- — F{Xj). Assume 
that n « yi- 

13. Determine a and b Io r which f(x) = asln{ji x/2) + bcns(nx/2) fits the following 
data in the least-squares sense. 


X 

-0.5 

-0.19 

0.02 

0.20 

0.35 

0.50 

y 

-3.558 

-2.874 

-1.995 

-1.040 

-0.068 

0.677 


14. Determine a and b so that f(x) = ax b fits the following data in the least-squares 
sense. 


X 

0.5 

1.0 

1.5 

2.0 

2.5 

y 

0.49 

1.60 

3.36 

6.44 

10.16 


15. Fit the function f(x) = axe bx to the data and compute the Standard deviation. 


X 

0.5 

1.0 

1.5 

2.0 

2.5 

y 

0.541 

0.398 

0.232 

0.106 

0.052 


16. ■ The intensity of radiation of a radioactive substance was measured at half-year 
intervals. The results were: 


t (years) 

0 

0.5 

1 

1.5 

2 

2.5 

Y 

1.000 

0.994 

0.990 

0.985 

0.979 

0.977 

t (years) 

3 

3.5 

4 

4.5 

5 

5.5 

Y 

0.972 

0.969 

0.967 

0.960 

0.956 

0.952 


where y is the relative intensity of radiation. Knowing that radioactivity decays 
exponentially with time: y{l) = ae~ bt , estimate the radioactive half-life of the 
substance. 


3.5 Other Methods 

Some data are better interpolated by rationalfunctions than by polynomials. Arational 
function R[x) is the quotient of two polynomials: 

_ Pmix) _ Op + a x x + a 2 x 2 H- a,rjc m 

x Q n (x) ly ) + b 1 x+ bzX 2 -\ - b„x n 
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3.5 Other Methods 


Since R(x) is a ratio, its coefficients can be scaled so that one of the coefficients (usually 
bo ) is unity. That leaves m+ n+ 1 undetermined coefficients which can be computed 
bypassing R{x) through m+ n+ 1 datapoints. 

The following data set is an ideal candidate for rational function interpolation: 


X 

0 

0.6 

0.8 

0.95 

y 

0 

1.3764 

3.0777 

12.7062 


From the plot of the data points (open circles in Fig. 3.8) it appears that y tends 
to infinity near x = 1. That makes the data ili suited for polynomial interpolation. 
Fiowever, the rational function interpolant 


/M 


1.3668X— 0.7515JC 2 
1 - 1.0013* 


(the solid line in Fig. 3.8) has no trouble handling the singularity. 



Figure 3.8. Rational function interpolation. 




















Roots of Equations 


Find the Solutions of /(x) = 0, where the function / is given 


4.1 Introduction 

A common problem encountered in engineering analysis is this: given a function /(x), 
determine the values of x for which /(x) = 0. The Solutions (values of x) are known as 
the roots of the equation /(x) = 0, or the zeroes of the function /(x). 

Before proceeding further, it might be helpful to review the concept of a function. 
The equation 

y= /w 

contains three elements: an input value x, an output value y, and the rule /for comput- 
ing y. The function is said to be given if the rule /is specified. In numerical computing 
the rule is invariably a computer algorithm. It may be a function statement, such as 

/(x) = cosh(x) cos(x) — 1 

or a complex procedure containing hundreds or thousands of lines of code. As long 
as the algorithm produces an output y for each input x, it qualifies as a function. 

The roots of equations may be real or complex. The complex roots are seldom 
computed, since they rarely have physical significance. An exception is the polynomial 
equation 

Oq + flix + aix 2 H-h a n x n = 0 

where the complex roots may be meaningful (as in the analysis of damped vibrations, 
for example). For the time being, we will concentrate on finding the real roots of 
equations. Complex zeroes of polynomials are treated near the end of this chapter. 
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4.2 Incremental Search Method 


In general, an equation may have any number of (real) roots, or no roots at ali. 
For example, 


sin x — x = 0 

has a single root, namely x = 0, whereas 

tan x - x = 0 

has an infinite number of roots (x = 0, ±4.493, ±7.725,...). 

All methods of finding roots are iterative procedures that require a starting point, 
i.e., an estimate of the root. This estimate can be crucial; a bad starting value may 
fail to converge, or it may converge to the “wrong” root (a root different from the one 
sought). There is no universal recipe for estimating the value of a root. If the equation is 
associated with a physical problem, then the context of the problem (physical insight) 
might suggest the approximate location of the root. Otherwise, a systematic numerical 
search for the roots can be carried out. One such search method is described in the 
next article. The roots can also be located visually by plotting the function. 

It is highly advisable to go a step further and bracket the root (determine its lower 
and upper bounds) before passing the problem to a root finding algorithm. Prior 
bracketing is, in fact, mandatory in the methods described in this chapter. 


4.2 Incremental Search Method 

The approximate locations of the roots are best determined by plotting the function. 
Often a very rough plot, based on a fewpoints, is sufficient to give us reasonable starting 
values. Another useful tool for detecting and bracketing roots is the incremental search 
method. It can also be adapted for computing roots, but the effort would not be 
worthwhile, since other methods described in this chapter are more efficient for that. 

The basic idea behind the incremental search method is simple: if /te) and /te) 
have opposite signs, then there is at least one root in the interval te. x 2 ). If the interval 
is small enough, it is likely to contain a single root. Thus the zeroes of /(x) can be 
detected by evaluating the function at intervals Ax and looking for change in sign. 
There are several potential problems with the incremental search method: 

• It is possible to miss two closely spaced roots if the search increment Ax is larger 
than the spacing of the roots. 

• A double root (two roots that coincide) will not be detected. 

• Certain singularities of /(x) can be mistaken for roots. For example, /(x) = tan x 
changes sign at x = ±\nix, n= 1,3, 5,..., as shown in Fig. 4.1. However, these 
locations are not true zeroes, since the function does not cross the x-axis. 
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Roots of Equations 



■ rootsearch 

This function searches for a zero of the user-supplied function f{x) in the interval 
(a, b) in increments of dx. It returns the bounds (xl, x2) of the root if the search 
was successful; xl = x2 = None indicates that no roots were detected. After the 
first root (the root closest to a) has been detected, rootsearch can be called again 
with a replaced by x2 in order to find the next root. This can be repeated as long as 
rootsearch detects a root. 


## module rootsearch 

’’’ xl,x2 = rootsearch(f,a,b,dx). 

Searches the interval (a,b) in increments dx for 
the bounds (xl,x2) of the smallest root of f(x). 
Returns xl = x2 = None if no roots were detected 

def rootsearch(f,a,b,dx): 
xl = a; fl = f(a) 
x2 = a + dx; f2 = f(x2) 
while fl*f2 > 0.0: 

if xl >= b: return None,None 
xl = x2; fl = f2 
x2 = xl + dx; f2 = f(x2) 
else : 

return xl,x2 


EXAMPLE 4.1 

Use incremental search with Ax = 0.2 to bracket the smallest positive zero of /(x) = 
x 3 — 10x 2 + 5. 
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4.3 Method of Bisection 


Solution We evaluate /(x) at intervals Ax = 0.2, staring at x = 0, until the function 
changes its sign (value of the function is of no interest to us; only its sign is relevant). 
This procedure yields the following results: 


X 

fw 

0.0 

5.000 

0.2 

4.608 

0.4 

3.464 

0.6 

1.616 

0.8 

-0.888 


From the sign change of the function we conclude that the smallest positive zero lies 
between x = 0.6 and x = 0.8. 


4.3 Method of Bisection 


After a root of /(x) = 0 has been bracketed in the interval (xi, x 2 ), several methods can 
be used to close in on it. The method of bisection accomplishes this by successively 
halving the interval until it becomes sufficiently small. This technique is also known 
as the interval halving method. Bisection is not the fastest method available for com- 
puting roots, but it is the most reliable. Once a root has been bracketed, bisection will 
always close in on it. 

The method of bisection uses the same principle as incremental search: if there 
is a root in the interval (xi, x 2 ), then /(xi) • /(x 2 ) < 0. In order to halve the interval, we 
compute /(x 3 ), where X 3 = \ [x\ + x 2 ) is the midpoint of the interval. If /(x 2 ) • /(x 3 ) < 
0 , then the root must be in (x 2 , x 3 ) and we record this by replacing the original bound 
Xi byx 3 . Otherwise, the root lies in (xi, X3), in which casex 2 is replacedbyx 3 . In either 
case, the new interval (xi, x 2 ) is half the size of the original interval. The bisection is 
repeated until the interval has been reduced to a small value e, so that 

1*2 — Xi| < e 

It is easy to compute the number of bisections required to reach a prescribed e. The 
original interval Ax is reduced to Ax/2 after one bisection, Ax/2 2 after two bisections, 
and after «bisections it is Ax/2". Setting Ax/2" = e and solving for n, we get 


ln (| Ax| /e) 

ln2 


(4.1) 


■ bisect 

This function uses the method of bisection to compute the root of f (x) = 0 that is 
known to lie in the interval (xl ,x2). The number of bisections n required to reduce 
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Roots of Equations 


the interval to tol is computed from Eq. (4.1). By setting switch = l, we force the 
routine to checkwhetherthe magnitude of f (x) decreases with each intervalhalving. 
If it does not, something may be wrong (probably the “root” is not a root at ali, but a 
singularity) and root = None is returned. Since this feature is not always desirable, 
the default value is switch = 0. The function error. err, which we use to terminate 
a program, is listed in Art. 1.6. 

## module bisect 

’’’ root = bisect(f,xl,x2,switch=0,tol=l.Oe-9) . 

Finds a root of f(x) = 0 by bisection. 

The root must be bracketed in (xl,x2). 

Setting switch = 1 returns root = None if 
f(x) increases as a resuit of a bisection. 

from math import log,ceil 
import error 

def bisect(f,xl,x2,switch=0,epsilon=l.0e-9): 
fl = f(xl) 

if f1 == 0.0: return xl 
f2 = f(x2) 

if f2 == 0.0: return x2 

if fl*f2 > 0.0: error.err(’Root is not bracketed’) 
n = ceil(log(abs(x2 - xl)/epsilon)/log(2.0)) 
for i in range(n): 

x3 = 0.5*(xl + x2); f3 = f(x3) 
if (switch == 1) and (abs(f3) >abs(fl)) \ 
and (abs(f3) > abs(f2)): 

return None 

if f3 == 0.0: return x3 
if f2*f3 <0.0: 

xl = x3; fl = f3 
else : 

x2 =x3; f2 = f3 
return (xl + x2)/2.0 

EXAMPLE 4.2 

Use bisection to find the root of /(x) = x 3 — 10x 2 + 5 = 0 that lies in the interval 
(0.6, 0.8). 

Solution The best way to implement the method is to use the following table. Note 
that the interval to be bisected is determined by the sign of /(x), not its magnitude. 
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4.3 Method of Bisection 


X 

/M 

Interval 

0.6 

1.616 

- 

0.8 

-0.888 

(0.6, 0.8) 

(0.6 + 0.8)/2 = 0.7 

0.443 

(0.7, 0.8) 

(0.8 + 0.7)/2 = 0.75 

-0.203 

(0.7, 0.75) 

(0.7 + 0.753/2 = 0.725 

0.125 

(0.725, 0.75) 

(0.75 + 0.7253/2 = 0.7375 

-0.038 

(0.725, 0.7375) 

(0.725 + 0.73753/2 = 0.73125 

0.044 

(0.7375, 0.73125) 

(0.7375 + 0.731253/2 = 0.73438 

0.003 

(0.7375, 0.73438) 

(0.7375 + 0.734383/2 = 0.73594 

-0.017 

(0.73438, 0.73594) 

(0.73438 + 0.735943/2 = 0.73516 

-0.007 

(0.73438, 0.73516) 

(0.73438 + 0.735163/2 = 0.73477 

-0.002 

(0.73438, 0.73477) 

(0.73438 + 0.734773/2 = 0.73458 

0.000 

- 


The final resuit x = 0.7346 is correct within four decimal places. 

EXAMPLE 4.3 

Find all the zeros of /( x) = x — tan x in the interval (0, 20) by the method of bisection. 
Utilize the functions rootsearch and bisect. 

Solution Note that tanx is singular and changes sign at x = tt/2, 3jt/2, _To pre- 

vent bisect from mistaking these point for roots, we set switch = l. The close- 
ness of roots to the singularities is another potential problem that can be alleviated 
by using small Ax in rootsearch. Choosing Ax= 0.01, we arrive at the following 
program: 

#!/usr/bin/python 
## example4_3 
from math import tan 
from rootsearch import * 
from bisect import * 

def f(x): return x - tan(x) 

a,b,dx = (0.0, 20.0, 0.01) 
print ’'The roots are:’’ 
while 1: 

xl,x2 = rootsearch(f,a,b,dx) 
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Roots of Equations 


if xl != None: 
a = x2 

root = bisect(f,xl,x2,1) 
if root != None: print root 
else : 

print ’’ \nDone ’’ 
break 

raw_input(’’Press return to exit’’) 

The output from the program is: 

The roots are: 

0.0 

4.4934094581 
7.72525183707 
10.9041216597 
14.0661939129 
17.2207552722 

Done 


4.4 Brent's Method 

Brent’s method 9 combines bisection and quadratic interpolation into an efficient 
root-finding algorithm. In most problems the method is much faster than bisection 
alone, but it can become sluggish if the function is not smooth. It is the recommended 
method of root solving if the derivative of the function is difficult or impossible to 
compute. 



9 Brent, R. P., Algorithms for Minimization without Derivatives, Prentice-Hall, 1973. 
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4.4 Brent's Method 


Brent's method assumes that a root of /(x) = 0 has been initially bracketed in 
the interval (xi, x 2 ). The root-finding process starts with a bisection step that halves 
the interval to either (xi, x 3 ) or (x 3 , x 2 ), vviiere x 3 = (xi + x 2 )/2, as showninFigs. 4.2(a) 
and (b). In the course of bisection we had to compute /1 = f(x\), f 2 = /(x 2 ) and 
f 3 = /(x 3 ), so that we now know three points on the /(x) curve (the open circles in 
the figure). These points allow us to carry out the next iteration of the root by inverse 
quadratio interpolation (viewing x as a quadratic function of /). If the resuit x of the 
interpolation falis inside the latest bracket (as is the case in Figs. 4.2) we accept the 
resuit. Otherwise, another round of bisection is applied. 




Figure 4.3. Relabeling roots after an iteration. 


The next step is to relabel x as x 3 and rename the limits of the new interval 
X\ and x 2 (x\ < x 3 < x 2 ), as indicated in Figs. 4.3. We have now recovered the orig- 
inal sequencing of points in Figs. 4.2, but the interval (xi, X2) containing the root 
has been reduced. This completes the first iteration cycle. In the next cycle an¬ 
other inverse quadratic interpolation is attempted and the process is repeated un- 
til the convergence criterion |x — x 3 | < e is satisfied, where e is a prescribed error 
tolerance. 

The inverse quadratic interpolation is carried out with Lagrange’s three-point 
interpolant described in Section 3.2. Interchanging the roles of x and /, we have 

rfn (/-/2K/-/3) , (/-/3) r , (/- fiKf- /2) r 

7 (/1 - /2K/1 - /3) 1 (/2 - /1K/2 - /3) 2 (/3 - /1K/3 - /2) 3 

Setting / = 0 and simplifying, we obtain for the estimate of the root 

/2 M (/2 - /3) + /3 M(/3 - /1) + /1M(/1 - /2) 
(/1-/2K/2-/3K/3-/1) 


The change in the root is 


Ax = x - x 3 = /3 


*3(/l - / 2)^/2 - /3 + / 1 ) + flXlifz - fi) + flX2if3 - / 1 ) 

“ /lK^( - ~ /3) 


(4.2) 
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Roots of Equations 


■ brent 

The function brent listed below is a simplified version of the algorithm proposed by 
Brent. It omits some of Brent’s safeguards against slow convergence; it also uses a less 
sophisticated convergence criterion. 

## module brent 

’’’ root = brent(f,a,b,tol=l.Oe-9) . 

Finds root of f(x) = 0 by combining quadratic interpolation 
with bisection (simplified Brent’s method). 

The root must be bracketed in (a,b). 

Calis user-supplied function f(x). 

import error 

def brent(f,a,b,tol=l.0e-9): 
xl = a; x2 = b; 
fl = f(xl) 

if f1 == 0.0: return xl 
f2 = f(x2) 

if f2 == 0.0: return x2 

if fl*f2 > 0.0: error.err(’Root is not bracketed’) 
x3 = 0.5*(a + b) 
for i in range(30): 
f3 = f(x3) 

if abs(f3) < tol: return x3 

# Tighten the brackets on the root 
if fl*f3 < 0.0: b = x3 

else: a = x3 

if (b - a) < tol*max(abs(b),1.0): return 0.5*(a + b) 

# Try quadratic interpolation 

denom = (f2 - fl)*(f3 - fl)*(f2 - f3) 
numer = x3*(fl - f2)*(f2 - f3 + fl) \ 

+ f2*xl*(f2 - f3) + fl*x2*(f3 - fl) 

# If division by zero, push x out of bounds 
try: dx = f3*numer/denom 

except ZeroDivisionError: dx = b - a 
x = x3 + dx 

# If iterpolation goes out of bounds, use bisection 
if (b - x)*(x - a) < 0.0: 
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4.4 Brent's Method 


dx = 0.5* (b - a) 
x = a + dx 

# Let x3 <-- x & choose new xl and x2 so that xl < x3 < x2 
if x < x3: 

x2 = x3; f 2 = f3 
else : 


xl = x3; fl = f3 
x3 = x 

print ’Too many iterations in brent’ 


EXAMPLE 4.4 

Determine the root of /(x) = x 3 - 10x 2 + 5 = 0 that lies in (0.6, 0.8) with Brent’s 
method. 

Solution 

Bisection The starting points are 

X\ = 0.6 f = 0.6 3 - 10(0.6) 2 + 5 = 1.616 

x 2 = 0.8 / 2 = 0.8 3 - 10(0.8) 2 + 5 = -0.888 

Bisection yields the point 

x 3 = 0.7 / 3 = 0.7 3 - 10(0.7) 2 + 5 = 0.443 

Byinspectingthesignsof/we conclude that the newbracketson the root are (x 3 , x 2 ) = 
(0.7, 0.8). 

First interpolation cycle The numerator of the quotient in Eq. (4.2) is 

num = x 3 (/i - / 2 )(/ 2 - / 3 + fi) + f 2 xi[f 2 - / 3 ) + /iX 2 (/ 3 - f) 

= 0.7(1.616 + 0.888)(—0.888 - 0.443 + 1.616) 

—0.888(0.6)(—0.888 - 0.443) + 1.616(0.8)(0.443 - 1.616) 

= -0.307 75 

and the denominator is 

den = (/ 2 - /i)(/ 3 - /i)(/ 2 - / 3 ) 

= (-0.888 - 1.616X0.443 - 1.616)(-0.888 - 0.443) = -3.9094 
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Therefore, 


Ax = 


num 

/3 d^T 


0.443 


(-0.307 75) 
(-3.9094) 


0.034 87 


and 


x = x 3 + Ax = 0.7 + 0.034 87 = 0.734 87 


Since the resuit is within the established brackets, we accept it. 


Relabel points As x > x 3 , the points are relabeled as illustrated in Figs. 4.2 (b) and 
4.3 (b); 

x 1 x 3 = 0.7 
/i <- /a = 0.443 
x 3 <— x = 0.734 87 

/ 3 = 0.734 87 3 - 10(0.734 87) 2 + 5 = -0.00348 
The new brackets on the root are (xi, x 3 ) = (0.7, 0.734 87) 


Secondinterpolation cycle ApplyingtheinterpolationinEq. (4.2) again, weobtain 
(skipping the arithmetical details) 


Ax = -0.000 27 

x = x 3 + Ax = 0.734 87 - 0.000 27 = 0.734 60 

Again x falis within the latest brackets, so the resuit is acceptable. At this stage, x is 
correct to five decimal places. 


EXAMPLE 4.5 

Compute the zero of 


/(x) = x|cosx| - 1 


that lies in the interval (0,4) with Brent's method. 
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4.5 Newton-Raphson Method 


Solution 



The plot of /( x) shows that this is a rather nasty function within the specified interval, 
containing a slope discontinuity and two local maxima. The sensible approach is 
to avoid the potentially troublesome regions of the function by bracketing the root 
as tightly as possible from a visual inspection of the plot. In this case, the interval 
(a, b) = (2.0, 2.2) would be a good starting point for Brent's algorithm. 

Is Brent’s method robust enough to handle the problem with the original brackets 
(0, 4)? Well, here is the program and its output: 

#!/usr/bin/python 
## example4_ 5 
from math import cos 
from brent import * 

def f(x): return x*abs(cos(x)) - 1.0 

print ’'root =’’,brent(f,0.0,4.0) 
raw_input(’’Press return to exit’’) 

root = 2.0739328091 

The resuit was obtained in only five iterations. 


4.5 Newton-Raphson Method 

The Newton-Raphson algorithm is the best-known method of finding roots for a 
good reason: it is simple and fast. The only drawback of the method is that it uses 
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the derivative /'(x) of the function as well as the function /(x) itself. Therefore, 
the Newton-Raphson method is usable only in problems where /'(x) can be readily 
computed. 

The Newton-Raphson formula can be derived from the Taylor series expansion 
of /(x) about x: 

f{x i+ 1 ) = fixi) + /'(x,0(x i+i - Xi) + 0(x i+ 1 - Xi) 2 (a) 

where O(z) is to be read as “of the order of z "—see Appendix AI. If x,- + i is a root of 
/(x) = 0, Eq. (a) becomes 

o = /(X;) + f'{Xi) (X i+ \ - Xi) + 0(X,-+1 - Xi) 2 (b) 


Assuming that x* is a close to x,-+i, we can drop the last term in Eq. (b) and solve for 
Xi + 1 . The resuit is the Newton-Raphson formula 


X/+1 = Xi - 


f(Xj) 

flXi) 


(4.3) 


If x denotes the true value of the root, the error in x,- is Ei = x — x,-. It can be shown 
that if Xj +1 is computed from Eq. (4.3), the corresponding error is 


f"{Xi) , 

E m = -±4Ae 2 


2/Tx- 


indicating that Newton-Raphson method converges quadratically (the error is the 
square of the error in the previous step). As a consequence, the number of signif¬ 
icant figures is roughly doubled in every iteration, provided that x,- is close to the 
root. 



Figure 4.4. Graphical interpretation of the Newton-Raphson 
formula. 


A graphical depiction of the Newton-Raphson formula is shown in Fig. 4.4. The for¬ 
mula approximates /(x) by the straight line that is tangent to the curve at x,-. Thus x,-+i 
is at the intersection of the x-axis and the tangent line. 

The algorithm for the Newton-Raphson method is simple: it repeatedly applies 
Eq. (4.3), starting with an initial value Xo, until the convergence criterion 

|Xj+i - Xi| < £ 

is reached, £ being the error tolerance. Only the latest value of x has to be stored. Here 
is the algorithm: 
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1. Let x be a guess for the root of f(x ) = 0. 

2. Compute Ax = — /(x)//'(x). 

3. Let x <- x + Ax and repeat steps 2-3 until | Ax| < e. 


m 


f(x) 


X 


(a) (b) 

Figure 4.5. Examples where the Newton-Raphson method 
diverges. 


Although the Newton-Raphson method converges fast near the root, its global 
convergence characteristics are poor. The reason is that the tangent line is not always 
an acceptable approximation of the function, as illustrated in the two examples in 
Fig. 4.5. But the method can be made nearly fail-safe by combining it with bisection, 
as in Brent’s method. 




■ newtonRaphson 

The following safe version of the Newton-Raphson method assumes that the root to 
be computed is initially bracketed in ( a , b ). The midpoint of the bracket is used as the 
initial guess of the root. The brackets are updated after each iteration. If a Newton- 
Raphson iteration does not stay within the brackets, it is disregarded and replaced 
with bisection. Since newtonRaphson uses the function f (x) as well as its derivative, 
function routines for both (denoted by f and df in the listing) must be provided by 
the user. 

## module newtonRaphson 

’’’ root = newtonRaphson(f,df,a,b,tol=l.Oe-9) . 

Finds a root of f(x) = 0 by combining the Newton-Raphson 
method with bisection. The root must be bracketed in (a,b). 
Calis user-supplied functions f(x) and its derivative df(x). 

def newtonRaphson(f,df,a,b,tol=1.0e-9): 
import error 
fa = f(a) 

if fa == 0.0: return a 
fb = f(b) 
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if fb == 0.0: return b 

if fa*fb > 0.0: error.err(’Root is not bracketed’) 
x = 0.5*(a + b) 
for i in range(30): 
fx = f(x) 

if abs(fx) < tol: return x 

# Tighten the brackets on the root 
if fa*fx < 0.0: 

b = x 
else : 

a = x; fa = fx 

# Try a Newton-Raphson step 
dfx = df(x) 

# If division by zero, push x out of bounds 
try: dx = -fx/dfx 

except ZeroDivisionError: dx = b - a 
x = x + dx 

# If the resuit is outside the brackets, use bisection 
if (b - x)*(x - a) < 0.0: 

dx = 0.5*(b-a) 
x = a + dx 

# Check for convergence 

if abs(dx) < tol*raax(abs(b),1.0): return x 
print 'Too many iterations in Newton-Raphson' 


EXAMPLE 4.6 

A root of /(x) = x 3 — 1 Ox 2 + 5 = 0 lies close to x = 0.7. Compute this root with the 
Newton-Raphson method. 


Solution The derivative of the function is /'(x) = 3x 2 — 20x, so that the Newton- 
Raphson formula in Eq. (4.3) is 

f{x) x 3 - 10x 2 + 5 2x 3 — 10x 2 - 5 

^ _ __ — _ _ — _ 

/'(x) 3x 2 — 20x x (3x - 20) 


It takes only two iterations to reach five decimal place accuracy: 

2(0.7) 3 — 10(0.7) 2 — 5 n _„„„ 

= 0.735 3b 


x <- 


0.7 [3(0.7) - 20] 

2(0.735 36) 3 - 10(0.735 36) 2 - 5 
0.735 36 [3(0.735 36) -20] 


= 0.734 60 
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EXAMPLE 4.7 

Find the smallest positive zero of 

f[x) = x 4 - 6.4x 3 + 6.45x 2 + 20.538x - 31.752 

Solutiori 



Inspecting the plot of the function, we suspect that the smallest positive zero is a 
double root near x = 2. Bisection and Brent’s method wouldnotwork here, since they 
depend on the function changing its sign at the root. The same argument applies to 
the function newtonRaphson. But there no reason why the unrefined version of the 
Newton-Raphson method should not succeed. We used the following program, which 
prints the number of iterations in addition to the root: 

#!/usr/bin/python 
## example4_ 7 

def f(x): return x**4 - 6.4*x**3 + 6.45*x**2 + 20.538*x - 31.752 
def df(x): return 4.0*x**3 - 19.2*x**2 + 12.9*x + 20.538 

def newtonRaphsonfx,tol=l.Oe-9) : 
for i in range(30): 
dx = -f(x)/df(x) 
x = x + dx 

if abs(dx) < tol: return x,i 
print ’Too many iterations\n’ 

root,numIter = newtonRaphson(2.0) 
print 'Root =’,root 

print 'Number of iterations =’,numlter 
raw_input(’’Press return to exit’’) 














158 


Roots of Equations 


The output is 

Root = 2.09999998403 
Number of iterations = 23 

The true value of the root is x = 2.1. It can be shown that near a multiple root the 
convergence of the Newton-Raphson method is linear, rather than quadratic, which 
explains the large number of iterations. Convergence to a multiple root can be speeded 
up by replacing the Newton-Raphson formula in Eq. (4.3) with 

Xj + i = Xi — m {} Xl ^ 

+ fiXi) 

where m is the multiplicity of the root (m = 2 in this problem). After making the change 
in the above program, we obtained the resuit in only 5 iterations. 


4.6 Systems of Equations 
Introduction 

Up to this point, we confined our attention to solving the single equation /(x) = 0. 
Let us now consider the rz-dimensional version of the same problem, namely 

f(x) = 0 


or, using scalar notation 


/l(Xi,X 2 . X„) = 0 

/ 2 (x 1 , x 2 ,..., x„) = 0 (4.4) 

/n(X|, X 2 , . . . , X n ) = 0 

The solution of n simultaneous, nonlinear equations is a much more formidable task 
than finding the root of a single equation. The trouble is the lack of a reliable method for 
bracketing the solution vector x. Therefore, we cannot provide the solution algorithm 
with a guaranteed good starting value of x, unless such a value is suggested by the 
physics of the problem. 

The simplest and the most effective means of computingxis the Newton-Raphson 
method. It works well with simultaneous equations, provided that it is supplied with 
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a good starting point. There are other methods that have better global convergence 
characteristics, but ali of then are variants of the Newton-Raphson method. 

Newton-Raphson Method 

In order to derive the Newton-Raphson method for a system of equations, we start 
with the Taylor series expansion of f (x) about the point x: 

n n f 

f(x + Ax) = f(x) + + 0(Ax 2 ) (4.5a) 

j =i dx J 

Dropping terms of order Ax 2 , we can write Eq. (4.5a) as 

f(x + Ax) = f(x) + J(x) Ax (4.5b) 

where J(x) is the Jacobian matrix (of size n x n) made up of the partial derivatives 

h > = Wj (46) 

Note that Eq. (4.5b) is a linear approximation (vector Ax being the variable) of the 
vector-valued function f in the vicinity of point x. 

Let us now assume that x is the current approximation of the solution of f(x) = 0, 
andletx+ Axbetheimprovedsolution.Tofindthecorrection Ax,wesetf(x+ Ax) = 0 
in Eq. (4.5b). The resuit is a set of linear equations for Ax: 

J(x)Ax = —f(x) (4.7) 

The following steps constitute the Newton-Raphson method for simultaneous, 
nonlinear equations: 

1. Estimate the solution vector x. 

2. Evaluate f(x). 

3. Compute the Jacobian matrix J(x) from Eq. (4.6). 

4. Set up the simultaneous equations in Eq. (4.7) and solve for Ax. 

5. Let x <- x + Ax and repeat steps 2-5. 

The above process is continued until | Ax| < e, where e is the error tolerance. As 
in the one-dimensional case, success of the Newton-Raphson procedure depends 
entirely on the initial estimate of x. If a good starting point is used, convergence to the 
solution is very rapid. Otherwise, the results are unpredictable. 

Because analytical derivation of each dfi/dXj can be difficult or impractical, it 
is preferable to let the computer calculate the partial derivatives from the finite 
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difference approximation 

H_ _ fiix+ejh) - fi[x) 
dXj h 

where h is a small increment and e ; - represents a unit vector in the direction of xj. 
This formula can be obtained from Eq. (4.5a) after dropping the terms of order Ax 2 
and setting Ax = ejh. We get away with the approximation in Eq. (4.8) because the 
Newton-Raphson method is rather insensitive to errors in J(x). By using this ap¬ 
proximation, we also avoid the tedium of typing the expressions for df/dXj into the 
computer code. 


■ newtonRaphson2 

This function is an implementation of the Newton-Raphson method. The nested 
function j acobian computes the Jacobian matrix from the finite difference approx¬ 
imation in Eq. (4.8). The simultaneous equations in Eq. (4.7) are solved by Gauss 
elimination with row pivoting using the function gaussPivot. listed in Section 2.5. 
The function subroutine f that returns the array f(x) must be supplied by the user. 

## module newtonRaphson2 

’’’ soln = newtonRaphson2(f,x,tol=l.0e-9). 

Solves the simultaneous equations f(x) = 0 by 
the Newton-Raphson method using {x} as the initial 
guess. Note that {f} and {x} are vectors. 

from numarray import zeros,Float64 , dot, sqrt 
from gaussPivot import * 

def newtonRaphson2(f,x,tol=1.0e-9) : 

def jacobian(f,x): 
h = 1.0e-4 
n = len(x) 

jac = zeros((n,n),type=Float64) 
fO = f(x) 

for i in range(n): 
temp = x[i] 
x[i] = temp + h 
fl = f(x) 
x[i] = temp 
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jac[: ,i] = (fl - fO)/h 
return jac,fO 

for i in range(30): 

jac,fO = jacobian(f,x) 

if sqrt(dot(fO,fO)/len(x)) < tol: return x 
dx = gaussPivot(jac,-fO) 
x = x + dx 

if sqrt(dot(dx,dx)) < tol*max(abs(x),1.0): return x 
print 'Too many iterations’ 

Note that the Jacobian matrix J (x) is recomputed in each iterative loop. Since each 
calculation of J(x) involves n + 1 evaluations of f(x) («is the numberof equations), the 
expense of computation can be high depending on «and the complexity of f(x). It is 
often possible to save computer time by neglecting the changes in the Jacobian matrix 
between iterations, thus computing J(x) only once. This will work provided that the 
initial x is sufficiently close to the solution. 

EXAMPLE 4.8 

Determine the points of intersection between the circle x 2 + y 2 = 3 and the hyperbola 
xy= 1. 

Solution The equations to be solved are 


Mx, y)=x 2 + f- 3 = 0 

(a) 

;< 

II 

* 

1 

I— 1 

II 

o 

(b) 


The Jacobian matrix is 


'dfi/dx dfi/dy 


2x 2 y 

_df 2 /dx df 2 /dy_ 


y x 


Thus the linear equations J(x)Ax=—f(x) associated with the Newton-Raphson 
method are 

2x 2 y 

y x 


Ax 


-x 2 - y 2 + 3 

A y 


—xy+ 1 


By plotting the circle and the hyperbola, we see that there are four points of 
intersection. It is sufficient, however, to find only one of these points, as the others 
can be deduced from symmetry. From the plot we also get a rough estimate of the 
coordinates of an intersection point: x = 0.5, y = 1.5, which we use as the starting 
values. 














162 


Roots of Equations 



The computations then proceed as follows. 


First iteration Substituting x = 0.5, y = 1.5 in Eq. (c), we get 


1.0 

3.o’ 

Ax 


0.50 

1.5 

0.5 

.Ay 


0.25 


the solution of which is Ax = A y = 0.125. Therefore, the improved coordinates ofthe 
intersection point are 


x = 0.5+ 0.125 = 0.625 y= 1.5 + 0.125= 1.625 

Second iteration Repeating the procedure using the latest values of x and y, we 
obtain 


1.250 

3.250" 

Ax 


—0.031250’ 

1.625 

0.625 

_Ay 


—0.015625 


which yields Ax = A y = —0.00694. Thus 

x = 0.625 -0.00694 = 0.61806 y= 1.625- 0.00694= 1.61806 

Third iteration Substitution of the latest x and y into Eq. (c) yields 


1.236 12 

3.23612’ 

Ax 


’—0.000116’ 

1.61806 

0.61806 

_Ay 


-0.000 058 


The solution is Ax = Ay = —0.00003, so that 

x = 0.618 06 - 0.000 03 = 0.618 03 
y = 1.618 06 - 0.000 03 = 1.618 03 

Subsequent iterations would not change the results within five significant figures. 
Therefore, the coordinates of the four intersection points are 

±(0.61803, 1.61803) and ± (1.61803, 0.61803) 
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Alternate solution If there are only a few equations, it may be possible to eliminate 
ali but one of the unknowns. Then we would be left with a single equation which can 
be solved by the methods described in Sections 4.2-4.5. In this problem, we obtain 
from Eq. (b) 



which upon substitution into Eq. (a) yields x 2 ± 1 /x 2 — 3 = 0, or 


x 4 - 3x 2 ± 1 = 0 


The Solutions of this biquadratic equation: x = ±0.618 03 and ±1.618 03 agree with 
the results obtained by the Newton-Raphson method. 

EXAMPLE 4.9 

Find a solution of 

sin x ± y 2 ± ln z — 7 = 0 
3x + 2 y — z? + 1 = 0 
x± y± z-5 = 0 

using newtonRaphson2. Start with the point (1,1,1). 

Solution Letting Xq = x, Xi = y and x 2 = z, we obtain the following program: 

#!/usr/bin/python 
## example4_9 

from numarray import zeros,array 
from math import sin.log 
from newtonRaphson2 import * 

def f(x): 

f = zeros((len(x)),type=Float64) 

f[0] = sin(x[0]) + x[l]**2 + log(x[2]) - 7.0 

f[l] = 3.0*x[0] + 2.0**x[l] - x[2]* * 3 + 1.0 

f[2] = x[0] + x[l] + x[2] - 5.0 

return f 

x = array([1.0, 1.0, 1.0]) 

print newtonRaphson2(f,x) 

raw_input (' '\nPress return to exit’’) 

The output from this program is 


[0.59905376 2.3959314 2.00501484] 
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PROBLEM SET4.1 

1. Use the Newton-Raphson method and a four-function calculator H— x-t oper- 
ations only) to compute -^75 with four significant figure accuracy. 

2. Find the smallest positive (real) root of x 3 — 3.23x 2 — 5.54x + 9.84 = 0 by the 
method of bisection. 

3. The smallest positive, nonzero root ofcoshx cos x — 1 = 01iesintheinterval(4, 5). 
Compute this root by Brent’s method. 

4. Solve Prob. 3 by the Newton-Raphson method. 

5. A root of the equation tan x — tanhx = 0 lies in (7.0, 7.4). Find this root with three 
decimal place accuracy by the method of bisection. 

6. Determine the two roots of sin x + 3 cos x — 2 = 0 that lie in the interval (—2,2). 
Use the Newton-Raphson method. 

7. A popular method in hand computation is the secantformulawhere the improved 
estimate of the root te+i) is obtained by linear interpolation based two previous 
estimates (jc,- and x,-_ i): 


Xi +1 = Xi - 


Xj ~ Xj-i 

/te) - /te-1) 


/te) 


Solve Prob. 6 using the secant formula. 

8. Draw a plot of /(x) = cosh x cos x — 1 in the range 4 < x < 8. (a) Verify from the 
plot that the smallest positive, nonzero root of /(x) = 0 lies in the interval (4, 5). 
(b) Show graphically that the Newton-Raphson formula would not converge to 
this root if it is started with x = 4. 

9. The equation x 3 - 1.2x 2 — 8.19x + 13.23 = 0 has a double root close to x = 2. 
Determine this root with the Newton-Raphson method within four decimal 
places. 

10. ■ Write a program that computes all the roots of /(x) = 0 in a given interval with 
Brent’s method. Utilize the functions rootsearch and brent. You may use the 
program in Example 4.3 as a model. Test the program by finding the roots of 
xsinx + 3 cosx — x = 0 in (—6, 6). 

11. ■ Solve Prob. 10 with the Newton-Raphson method. 

12. ■ Determine all real roots of x 4 + 0.9x 3 — 2.3x 2 + 3.6x — 25.2 = 0. 

13. ■ Compute all positive real roots of x 4 + 2x 3 - 7x 2 + 3 = 0. 

14. ■ Find all positive, nonzero roots of sin x — 0. lx = 0. 
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15. ■ The natural frequencies of a uniform cantilever beam are related to the roots 
Pi of the frequency equation /(/J) = cosh p cos (6 + 1 = 0, where 


(2jt fi) 2 


ml? 

EI 


fi = zth natural frequency (cps) 
m = mass of the beam 
L = length of the beam 
E = modulus of elasticity 
I = moment of inertia of the cross section 


Determine the lowest two frequencies of a Steel beam 0.9 m. long, with a rectan- 
gular cross section 25 mm wide and 2.5 mm in. high. The mass density of Steel is 
7850 kg/m 3 and E = 200 GPa. 

16. ■ 

L L 

_ _ i _ _ _2^_^ 

Length =s ' 'O 


A Steel cable of length s is suspended as shown in the figure. The maximum tensile 
stress in the cable, which occurs at the supports, is 

Umax = Uq COSh /3 


where 


v L 
B = — 

2u o 

g o = tensile stress in the cable at O 
Y = weight of the cable per unit volume 
L = horizontal span of the cable 


The length to span ratio of the cable is related to /3 by 

s 1 . , 

— = — smh p 
L p 

Find o ma x if y = 77 x 10 3 N/m 3 (steel), L = 1000 m and s = 1100 m. 
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The aluminum W310 x 202 (wideflange) columnissubjectedtoaneccentricaxial 
load P as shown. The maximum compressive stress in the column is given by the 
so-called secant formula: 


G max — & 


n eC 

1 + ^ sec 
r l 




where 


<7 = P/ A = average stress 

A = 25 800 mm 2 = cross-sectional area of the column 
e = 85 mm = eccentricity of the load 
c = 170 mm = half-depth of the column 
r = 142 mm = radius of gyration of the cross section 
L = 7100 mm = length of the column 
£=71xl0 9 Pa = modulus of elasticity 

Determine the maximum load P that the column can carry if the maximum stress 
is not to exceed 120 x 10 6 Pa. 

18. ■ 



Bernoulli’s equation for fluid flow in an open channel with a small bump is 

Q 2 , , Q 2 


2 g#K 


+ /Jo = 


2 gW-W 


+ h+H 


where 


Q = 1.2 m 3 /s = volume rate of flow 
g = 9.81 m/s 2 = gravitational acceleration 
b = 1.8 m = width of channel 
/to = 0.6 m = upstream water level 
H = 0.075 m = height of bump 
h = water level above the bump 


Determine h. 
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19. ■ The speed v of a Satum V rocket in vertical flight near the surface of earth can 
be approximated by 


v= uln 


Mo 

Mq - mt 


gt 


where 


u = 2510 m/s = velocity of exhaust relative to the rocket 
Mq = 2.8 x 10 6 kg = mass of rocket at liftoff 
m= 13.3 x 10 3 kg/s = rate offuel consumption 
g=9.81m/s 2 = gravitational acceleration 
t = time measured from liftoff 

Determine the time when the rocket reaches the speed of sound (335 m/s). 

20. ■ 


Heating at 
constant \ 


Isothermal 

expansion 



The figure shows the thermodynamic cycle of an engine. The efficiency of this 
engine for monoatomic gas is 


in(r 2 /?j)-(i-7i/r 2 ) 

11 ln(r 2 /71) + (l-71/r 2 )/(y-l) 

where T is the absolute temperature and y = 5/3. Find T 2 /T\ that results in 30% 
efficiency (^ = 0.3). 

21. ■ Gibb’s free energy of one mole of hydrogen at temperature T is 

G = -RT\n[{T/Tof /2 ] J 

where R = 8.314 41 J/K is the gas constant and T 0 = 4.444 18 K. Determine the 
temperature at which G = —10 5 J. 
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22. ■ The Chemical equilibrium equation in the production of methanol from CO 
and H 2 is 10 


?(3-2f) 2 

( l -?) 3 


249.2 


where f is the equilibrium extent ofthe reaction. Determine £. 

23. ■ Determine the coordinates of the two points where the circles (x — 2) 2 + y 2 = 4 
andx 2 + (y- 3) 2 = 4 intersect. Startby estimating the locations ofthe points from 
a sketch of the circles, and then use the Newton-Raphson method to compute 
the coordinates. 


24. ■ The equations 


sin x + 3 cos x — 2 = 0 
cos x — sin y+ 0.2 = 0 

have a solution in the vicinity of the point (1,1). Use the Newton-Raphson method 
to refine the solution. 

25. ■ Use any method to find ali real Solutions in 0 < x < 1.5 of the simultaneous 
equations 


tan x — y = 1 
cosx — 3siny = 0 

26. ■ The equation of a circle is 

{x-af + ( y-b) 2 = R 2 


where R is the radius and ( a, b ) are the coordinates of the center. If the coordinates 
of three points on the circle are 


X 

8.21 

0.34 

5.96 

y 

0.00 

6.62 

-1.12 


determine R, a and b. 

27. ■ 



10 FromAlberty, R. A., Physical Chemistry, 7th ed., Wiley, 1987. 
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The trajectory of a satellite orbiting the earth is 


1 + e sin(0 + a) 


where ( R, 0) are the polar coordinates of the satellite, and C, e and a are constants 
(e is known as the eccentricity of the orbit). If the satellite was observed at the 
following three positions 


9 

-30° 

0° 

O 

O 

cn 

R (km) 

6870 

6728 

6615 


determine the smallest R of the trajectory and the corresponding value of 0. 

28. ■ 


X 

A projectile is launched at O with the velocity v at the angle 0 to the horizontal. 
The parametric equations of the trajectory are 

x = (vcos0)f 

1 9 

y= --gr + (ysin«)t 

where fis the time measured from the instant of launch, and g = 9.81 m/s 2 rep- 
resents the gravitational acceleration. If the projectile is to hit the target at the 45° 
angle shown in the figure, determine v, 9 and the time of flight. 

29. ■ 



The three angles shown in the figure of the four-bar linkage are related by 
150 cos 0i + 180 cos 0 2 — 200 cos 0 3 = 200 
15Osin0i + 18Osin0 2 — 2OOsin0 3 = 0 
Determine 9 1 and 9 2 when 0 3 = 75°. Note that there are two Solutions. 
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*4.7 Zeroes of Polynomials 
Introduction 

A polynomial of degree nhas the form 


P n {x) = cto + aix + a 2 x 2 -I-1- a n x n 


(4.9) 


where the coefficients a,- may be real or complex. We will concentrate on polynomials 
with real coefficients, but the algorithms presented in this chapter also work with 
complex coefficients. 

The polynomial equation P n (x) = 0 has exactly n roots, which may be real or 
complex. If the coefficients are real, the complex roots always occur in conjugate pairs 
( x r + ixi, x r — ixi), where x r and x,- are the real and imaginary parts, respectively. For 
real coefficients, the number of real roots can be estimated from the rule ofDescartes: 

• The number of positive, real roots equals the number of sign changes in the 
expression for P n (x), or less by an even number. 

• The number of negative, real roots is equal to the number of sign changes in 
P n {—x), or less by an even number. 

As an example, consider P 3 (x) = x 3 — 2x 2 — 8x + 27. Since the sign changes twice, 
P’ (x) = 0 has either two or zero positive real roots. On the other hand, P\{—x) = 
—x 3 — 2x 2 + 8x + 27 contains a single sign change; hence P 3 (x) possesses one negative 
real zero. 

The real zeros of polynomials with real coefficients can always be computed by 
one of the methods already described. But if complex roots are to be computed, it is 
best to use a method that specializes in polynomials. Here we present a method due to 
Laguerre, which is reliable and simple to implement. Before proceeding to Laguerre’s 
method, we must first develop two numerical tools that are needed in any method 
capable of determining the zeroes of a polynomial. The first of these is an efficient 
algorithm for evaluating a polynomial and its derivatives. The second algorithm we 
need is for the deflation of a polynomial, i.e., for dividing the P„(x) by x — r, where r is 
a root of P n [x) = 0. 

Evaluation of Polynomials 

It is tempting to evaluate the polynomial in Eq. (4.9) from left to right by the following 
algorithm (we assume that the coefficients are stored in the array a): 

p = 0.0 

for i in range(n+l): 
p = p + a[i]*x**i 
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Since x k is evaluated asxxxx---xx(fc — 1 multiplications), we deduce that the 
number of multiplications in this algorithm is 

1 

1 + 2 + 3H- \-n — 1 = - «(« — ^ 

If «is large, the number of multiplications can be reduced considerably ifwe evaluate 
the polynomial from right to left. For an example, take 

P 4 (x) = do + d\X + a 2 X 2 + OgX 3 + CI 4 X 4 
After rewriting the polynomial as 

J° 4 (x) = cio + x [ai + x [a 2 + x (« 3 + xaj)]} 
the preferred computational sequence becomes obvious: 

P 0 (x) = 04 
Pl(x) = 03 + xP 0 (x) 

P 2 {X) = a 2 + xP t (x) 

P 3 (x) = «1 + xP 2 (x) 

P 4 (x) = Oq + xP 3 (x) 

For a polynomial of degree n, the procedure can be summarized as 
PoU) = a n 

PiW = On-i + xP ; _i (x), i=l,2,...,n (4.10) 

leading to the algorithm 
P = a[n] 

for i in range(1,n+1): 
p = a[n-i] + p*x 

The last algorithm involves only n multiplications, making it more efficient for 
n > 3. But computational economy is not the prime reason why this algorithm should 
be used. Because the resuit of each multiplication is rounded off, the procedure with 
the least number of multiplications invariably accumulates the smallest roundoff 
error. 

Some root-finding algorithms, including Laguerre’s method, also require evalua- 
tion of the first and second derivatives of P„(x). From Eq. (4.10) we obtain by differ- 
entiation 


Pq(x) = 0 i*(x) = Pi_i(x) + x^_i(x), i=l,2,...,n 

Po (x) = 0 P"(x) = 2P[_ X (x) + xP”_ x (x), 1=1,2,...,« 


(4.11 a) 
(4.1 lb) 
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■ evalPoly 

Here is the function that evaluates a polynomial and its derivatives: 

## module evalPoly 

’’’ p,dp,ddp = evalPolyCa,x). 

Evaluates the polynomial 

p = a[0] + a[l]*x + a[2]*x~2 +...+ a[n]*x"n 
with its derivatives dp = p’ and ddp = p'' 
at x. 

def evalPolyCa,x): 
n = len(a) - 1 
P = a[n] 
dp = 0.0 + O.Oj 
ddp = 0.0 + O.Oj 
for i in range(1,n+1): 

ddp = ddp*x + 2.0*dp 
dp = dp*x + p 
p = p*x + a[n-i] 
return p,dp,ddp 


Deflation of Polynomials 

After a root r of P„[x) = 0 has been computed, it is desirable to factor the polynomial 
as follows: 

P n (x) = (x - r) P n -1 (x) (4.12) 

This procedure, known as deflation or synthetic divisiori, involves nothing more than 
computing the coefficients of /V i (x) . Since the remaining zeros of P,,{x) are also the 
zeros of P„_i (x) , the root-finding procedure can nowbe applied to P n ~\ (x) rather than 
P„[x). Deflation thus makes it progressively easier to find successive roots, because 
the degree of the polynomial is reduced every time a root is found. Moreover, by 
eliminating the roots that have already been found, the chances of computing the 
same root more than once are eliminated. 

If we let 

P„- i(x) = ba + bix+ bzx 2 4-1- b n - ix ,i_1 

thenEq. (4.12) becomes 

Oq + a\X + a 2 x 2 4-1- a n - 1 x n ~ 1 + a n x n 

= (x - r) (h) + hx + hx 2 4-h b n ~\x n ~ l ) 
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4.7 Zeroes of Polynomials 


Equating the coefficients of like powers of x, we obtain 


b n -1 = a n b n - 2 = a n -i + rb n -i ■ ■ • b 0 = a l + rb L 


(4.13) 


which leads to the Horner’s deflation algorithm: 
b[n-l] = a[n] 

for i in range(n-2,-1,-1): 
b[i] = a[i+l] + r*b[i+l] 

Laguerre's Method 

Laguerre’s formulas are not easily derived for a general polynomial P„(x ). However, 
the derivation is greatly simplified if we consider the special case where the poly¬ 
nomial has a zero at x = r and ( n — 1) zeros at x = q. Hence the polynomial can be 
written as 


P n {x) = (x- r)(x - q) n 1 


(a) 


Our problem is now this: given the polynomial in Eq. (a) in the form 
P„(x) = cio + aix + a 2 x 2 -|-b OnX 11 


determine r (note that q is also unknown). It turns out that the resuit, which is ex- 
act for the special case considered here, works well as an iterative formula with any 
polynomial. 

Differentiating Eq. (a) with respect to x, we get 


P'(x) = {x - q) n -' + (n - l)(x - r){x - q) n ~ 2 



Thus 


P'n (X) 1 n- 1 

-irrr = -1- 


P„(x) x-r x-q 


+ 


(b) 


which upon differentiation yields 


P"(x) [F„(i)1 2 _ 1 n- 1 


(c) 


Pnix) L p nM J (x - r) 2 (x - q) 2 


It is convenient to introduce the notation 



Pnix) 


H{x ) = G 2 (x) 


(4.14) 
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so that Eqs. (b) and (c) become 


G(x) 


1 n- 1 

- 1 - 

x - r x — q 


H{x) = 


1 n- 1 

(x - r) 2 + (x - q) 2 


(4.15a) 

(4.15b) 


If we solve Eq. (4.15a) for x - q and substitute the resuit into Eq. (4.15b), we obtain a 
quadratic equation for x - r. The solution of this equation is the Laguerre’s formula 

n 

x—r= -(4.16) 

G(x) ± 1) [nH{x) - G 2 (x)] 


The procedure for finding a zero of a general polynomial by Laguerre’s formula is: 


1. Let x be a guess for the root of P n {x) = 0 (any value will do). 

2. Evaluate P„(x), P' n {x) and P"(x) using the procedure outlined in Eqs. (4.10) and 
(4.11). 

3. Compute G(x) and H(x) from Eqs. (4.14). 

4. Determine the improved root r from Eq. (4.16) choosing the sign that results 
in the larger magnitude of the denominator (this can be shown to improve 
convergence). 

5. Let x •<- r and repeat steps 2-5 until | P n (x) \ < e or |x — r| < e, where e is the error 
tolerance. 


One nice property of Laguerre’s method is that it converges to a root, with very 
few exceptions, from any starting value of x. 


■ polyRoots 

The function polyRoots in this module computes all the roots of P n [x) = 0, where 
the polynomial P n (x) defined by its coefficient array a = [<%, a\ ,..., a n \ . After the first 
root is computed by the nested function laguerre, the polynomial is deflated using 
def lPoly and the next zero computed by applying laguerre to the deflated polyno¬ 
mial. This process is repeated until all n roots have been found. If a computed root has 
a very small imaginary part, it is very likely that it represents roundoff error. Therefore, 
polyRoot s replaces a tiny imaginary part by zero. 

## module polyRoots 
’ ’ ' roots = polyRoots(a). 

Uses Laguerre’s method to compute all the roots of 
a[0] + a[l]*x + a[2]*x~2 +...+ a[n]*x~n = 0. 

The roots are returned in the vector {roots}, 
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from evalPoly import * 
from numarray import zeros,Complex64 
from cmath import sqrt 
from random import random 

def polyRoots(a,tol=l.Oe-12) : 

def laguerre(a,tol): 

x = randomO # Starting value (random number) 

n = len(a) - 1 

for i in range(30): 

p,dp,ddp = evalPoly(a,x) 

if abs(p) < tol: return x 

g = dp/p 

h = g*g - ddp/p 

f = sqrt((n - l)*(n*h - g*g)) 

if abs(g + f) > abs(g - f): dx = n/(g + f) 

else: dx = n/(g - f) 

x = x - dx 

if abs(dx) < tol*max(abs(x),1.0): return x 
print ’Too many iterations in Laguerre’ 

def deflPoly(a,root): # Deflates a polynomial 

n = len(a)-l 
b = [(0.0 + 0.0j)]*n 
b[n-l] = a[n] 

for i in range(n-2,-1,-1): 

b[i] = a[i+l] + root*b[i+l] 
return b 

n = len(a) - 1 

roots = zeros((n),type=Complex64) 
for i in range(n): 

x = laguerre(a,tol) 
if abs(x.imag) < tol: x = x.real 
roots[i] = x 
a = deflPoly(a,x) 
return roots 

raw_input('’ \nPress return to exit’’) 
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Since the roots are computed with finite accuracy, each deflation introduces small 
errors in the coefficients of the deflated polynomial. The accumulated roundoff error 
increases with the degree of the polynomial and can become severe if the polynomial is 
ill-conditioned (small changes in the coefficients produce large changes in the roots). 
Hence the results should be viewed with caution when dealing with polynomials of 
high degree. 

The errors caused by deflation can be reduced by recomputing each root using 
the original, undeflated polynomial. The roots obtained previously in conjunction 
with deflation are employed as the starting values. 


EXAMPLE 4.10 

A zero of the polynomial P 4 (x) = 3x 4 — 1 Ox 3 — 48x 2 — 2x + 12 is x = 6 . Deflate the 
polynomial with fforner’s algorithm, i.e., find P 3 (x) so that (x — 6 ) P 3 (x) = P 4 (x). 


Solution With r = 6 and n = 4, Eqs. (4.13) become 

bi = O} = 3 

bz — = — 10 —1— 6(3) = 8 

b\ — U 2 ~\~ 6 bz — —48 T 6(8) = 0 
bo = ci\ T Gb\ — —2 + 6(0) = —2 


Therefore, 


P 3 (x) = 3x 3 + 8 x 2 - 2 


EXAMPLE 4.11 

A root of the equation P 3 (x) = x 3 — 4.0x 2 — 4.48x + 26.1 is approximately x = 3 — i. 
Find a more accurate value of this root by one application of Laguerre’s iterative 
formula. 


Solution Use the given estimate of the root as the starting value. Thus 
x = 3 — Z x 2 = 8 — 6 i x 3 = 18 - 26/ 


Substituting these values in ftlx) and its derivatives, we get 
P 3 (x) = x 3 — 4.0X 2 — 4.48x + 26.1 

= (18 - 26/) - 4.0(8 - 6 /) - 4.48(3 - i) + 26.1 = -1.34 + 2.48/ 
P 3 M = 3-Ox 2 - 8.0x - 4.48 

= 3.0(8 - 6 /) - 8.0(3 - Z) - 4.48 = -4.48 - 10.0/ 
i^'(x) = 6.0x - 8.0 = 6.0(3 - /) - 8.0 = 10.0 - 6.0/ 
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Equations (4.14) thenyield 

F 3 { x) -4.48-10.0/ 


G(x) = 


P 3 (x) -1.34 + 2.48/ 


-2.36557 + 3.08462/ 


p" (y) 

H{x) = G 2 (x) - = (-2.36557+ 3.08462/) 2 

P 3 {x) 

= 0.35995 - 12.48452/ 


10.0 - 6 . 0 / 
-1.34 + 2.48/ 


The term under the square root sign of the denominator in Eq. (4.16) becomes 
F(x) = y]{n- 1) [nH(x) - G 2 (x)] 

= yjl [3(0.35995 - 12.48452/) - (-2.36557 + 3.08462/) 2 ] 

= V5.67822 - 45.71946/ = 5.08670 - 4.49402/ 

Now we must find which sign in Eq. (4.16) produces the larger magnitude of the 
denominator: 


|G(x) + F(x)\ = | (-2.36557 + 3.08462/) + (5.08670 - 4.49402/)| 

= 12.72113 - 1.40940/] = 3.06448 
|G(x) - F(x)\ = |(-2.36557 + 3.08462/) - (5.08670 - 4.49402/)| 

= 1-7.45227 + 7.57864/1 = 10.62884 

Using the minus sign, we obtain from Eq. (4.16) the following improved approximation 
for the root 


G(x) - F{x) -7.45227 + 7.57864/ 

= 3.19790 - 0.79875/ 

Thanks to the good starting value, this approximation is already quite close to the 
exact value r = 3.20 — 0.80/. 


EXAMPLE 4.12 

Use polyRoots to compute all the roots of x 4 — 5X 3 - 9x 2 + 155x — 250 = 0. 
Solution The commands 


»> from polyRoots import * 

»> print polyRoots( [-250.0,155.0 ,-9.0 , - 5.0,1.0] ) 
resulted in the output 
[2,+O.j 4.-3.j 4.+3.J -5.+0.j] 
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PROBLEM SET4.2 

Problems 1-5 A zero x = r of P„(x) is given. Verify that r is indeed a zero, and then 
deflate the polynomial, i.e., find P„_i (x) so that P n (x) = (x — r) P„_i (x). 

1. Pj(x) = 3X 3 + 7x 2 — 36x + 20, r = —5. 

2. P 4 (x) = x 4 - 3x 2 + 3x - 1 , r = 1. 

3. P 5 (x) = x 5 - 30x 4 + 36lx 3 - 2178x 2 + 6588x - 7992, r = 6 . 

4. P 4 (x) = x 4 — 5x 3 — 2x 2 — 20x — 24, r = 2 i. 

5. P 3 (x) = 3x 3 - 19x 2 + 45x - 13, r = 3 - 2 i. 

Problems 6-9 A zero x = r of P n (x) is given. Determine ali the other zeros of P„(x) 
by using a calculator. You should need no tools other than deflation and the quadratic 
formula. 

6 . P 3 (x) = x 3 + 1.8x 2 - 9.01x - 13.398, r = -3.3. 

7. P 3 (x) = x 3 - 6.64x 2 + 16.84x - 8.32, r = 0.64. 

8 . P 3 (x) =2x 3 - 13x 2 + 32x - 13, r = 3 - 2 i. 

9. Pj(x) = x 4 — 3x 3 + 10x 2 — 6 x — 20, r = 1 + 3 i. 

Problems 10-15 Find ali the zeros of the given P„(x). 

10. BP 4 (x) = x 4 + 2.1x 3 - 2.52x 2 + 2.1x - 3.52. 

11. ■ P 5 (x) = x 5 — 156x 4 - 5x 3 + 780x 2 + 4x - 624. 

12. BPelx) = x 6 + 4x 5 - 8 x 4 - 34x 3 + 57x 2 + 130x - 150. 

13. BP 7 (x) = 8 x 7 + 28x 6 + 34x 5 - 13x 4 - 124x 3 + 19x 2 + 220x - 100. 

14. ■ P 3 (x) = x 8 - 7x 7 + 7x 6 + 25x 5 + 24x 4 - 98x 3 - 472x 2 + 440x + 800. 

15. ■ P 4 (x) = x 4 + (5 + i)x 3 - (8 - 5i)x 2 + (30 - 14i)x - 84. 

16. ■ 
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4.8 Other Methods 


The two blocks of mass m each are connected by springs and a dashpot. The 
stiffness of each spring is k, and c is the coefficient of damping of the dashpot. 
When the system is displaced and released, the displacement of each block during 
the ensuing motion has the form 


x k {t) = cos{a>it + (f> k ), k= 1, 2 


where A k and <j> k are constants, and a> = co r ± ia>i are the roots of 



Determine the two possible combinations of &> r and if c/ m= 12 s 1 and kj m= 


1500 s —2 . 


4.8 Other Methods 


The most prominent root-finding algorithms omitted from this chapter are the secant 
method and its close relative, the false position method. Both methods compute the 
improved value of the root by linear interpolation. They differ only by how they choose 
the points involved in the interpolation. The secant method always uses the two most 
recent estimates of the root, whereas the false position method employs the points that 
keep the root bracketed. The secant method is faster of the two, but the false position 
method is more stable. Since both are considerably slower than Brent’s method, there 
is little reason to use them. 

There are many methods for finding zeros of polynomials. Of these, the Jenkins- 
Traub algorithm 11 deserves special mention due to its robustness and widespread use 
in packaged Software. 

The zeros of a polynomial can also be obtained by calculating the eigenvalues of 
the n x n “companion matrix” 


a„- 11 a n ci'i / a tj 


ci\ j a n 0 (j j a n 


1 0 


0 0 
0 0 


A = 


0 


1 


0 


0 


1 


0 


11 


Jenkins, M., and Traub, J., SIAMJournal on Numerical Analysis, Vol. 7 (1970), p. 545. 
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where are the coefficients of the polynomial. The characteristic equation (see 
Section 9.1) of this matrix is 

n i ti— i , dri—2 n—2 , ^1 P) r , 

X -\ - X H- X H-1-X H-=0 

& n &n. 

which is equivalent to P n [x ) = 0. Thus the eigenvalues of A are the zeroes of P„{x). The 
eigenvalue method is robust, but considerably slower than Laguerre’s method. But it 
is worthy of consideratiori if a good program for eigenvalue problems is available. 
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Numerical Differentiation 


Given the function /(x), compute d n f/dx n at given x 


5.1 Introduction 

Numerical differentiation deals with the following problem: we are given the function 
y = f(x) and wish to obtain one of its derivatives at the point x = x*. The term “given” 
means that we either have an algorithm for computing the function, or possess a 
set of discrete data points (x,, yd, i = 0,1, ...,n In either case, we have access to a 
finite number of (x, y) data pairs from which to compute the derivative. If you suspect 
by now that numerical differentiation is related to interpolation, you are right—one 
means of finding the derivative is to approximate the function locally by a polynomial 
and then differentiate it. An equally effective tool is the Taylor series expansion of /(x) 
about the point Xk, which has the advantage of providing us with information about 
the error involved in the approximation. 

Numerical differentiation is not a particularly accurate process. It suffers from 
a conflict between roundoff errors (due to limited machine precision) and errors 
inherent in interpolation. For this reason, a derivative of a function can never be 
computed with the same precision as the function itself. 


5.2 Finite Difference Approximations 

The derivation of the finite difference approximations for the derivatives of /(x) is 
based on forward and backward Taylor series expansions of /(x) about x, such as 
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Numerical Differentiation 


/t 2 /z 3 U 

f{x + h) = /(x) + hf(x) + — /"(x) + — /'"(x) + — / (4) (x) 4 - (a) 

/z 2 /z 3 /Z* 

/(X - /Z) = /(X) - /z/ (x) + - f\x) - - /'"(x) + - / l4) (x) - ■ • • (b) 

/(x + 2/z) = /(x) + 2/z/'(x) + f" (x) + /'"(x) + / (4) (x) H- (c) 

/(x -2 h)= /(x) - 2hf'{x) + f (x) - /'"(x) + / (4) (x)- (d) 


We also record the sums and differences of the series: 


/(x + /z) + /(x - h) =2 /(x) + /z 2 f (x) + ^ / (4) (x) H- 

(e) 

/z 3 

/(x + /z) - /(x - li) = 2hf\x) + — /'"(x) d- 

(f) 

4fz 4 

f[x+2h) + /(x- 2/z) = 2/(x) + 4/z 2 /"(x) + —/ 4 ) (x) H- 

(g) 

8 /z 3 

/(x + 2/z) - /(x - 2/z) = 4hf[x) + — /"'(x) H- 

(h) 

Note that the sums contain only even derivatives, whereas the differences retain just 
the odd derivatives. Equations (a)-(h) can be viewed as simultaneous equations that 
can be solved for various derivatives of /(x). The number of equations involved and 
the number of terms kept in each equation depend on the order of the derivative and 
the desired degree of accuracy. 

First Central Difference Approximations 

The solution of Eq. (f) for /'(x) is 


or 

(5.1) 


which is called die first Central difference approximation for /'(x). The term (!)(/z 2 ) 
reminds us that the truncation error behaves as /z 2 . 
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5.2 Finite Difference Approximations 


Similarly, Eq. (e) yields the first Central difference approximation for /" (x): 

ru> = /<*+«- 2 /w+/<*-» + 

or 

fw= /tt + W-2^ ) + /te-' 1) + 0 , tfl (52) 

Central difference approximations for other derivatives can be obtained from 
Eqs. (a)-(h) in the same manner. For example, eliminating /'(x) from Eqs. (f) and (h) 
and solving for f"'{x ) yield 

rui - 2 '-) + ovf) (5 3) 


The approximation 

y ( 4 ) (JC) = /U + 2ft) - 4/(x + h) + 6/(x) - 4/(x - h) + /(x - 2fr) + 4) 

is available from Eqs. (e) and (g) after eliminating /"(x). Table 5.1 summarizes the 
results. 



/(x — 2 h) 

/(x - h) 

/M 

/(x + h ) 

f(x + 2h) 

2 hf'[x) 


-1 

0 

1 


/z 2 f"[x) 


1 

-2 

1 


2 tff'"{x) 

-1 

2 

0 

-2 

1 

h' /' (4) (x) 

1 

-4 

6 

-4 

1 


Table 5.1. Coefficients of Central finite difference approximations 
of 


First Noncentral Finite Difference Approximations 

Central finite difference approximations are not always usable. For example, con- 
sider the situation where the function is given at the n discrete points x 0 . X|...., x„. 
Since Central differences use values of the function on each side of x, we would 
be unable to compute the derivatives at Xq and x n . Clearly, there is a need for fi¬ 
nite difference expressions that require evaluations of the function only on one 
side of x. These expressions are called forward and backward finite difference 
approximations. 
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Noncentral finite differences can also be obtained from Eqs. (a)-(h). Solving Eq. (a) 
for /'(x) we get 


f M 


/(x + h) - /(x) 
h 


h 

frw--rw 


4l" 


/ (4 , (x)--- 


Keeping only the first term on the right-hand side leads to the firstforward difference 
approximation 


f'(x) 


/(x + h) - /(x) 
h 


+ om 


(5.5) 


Similarly, Eq. (b) yields th e first backward difference approximation 


f M 


/M 


/(x - h) 
h 


+ O01) 


(5.6) 


Note that the truncation error is now 0[h), which is not as good as O ( If ) in Central 
difference approximations. 

We can derive the approximations for higher derivatives in the same manner. For 
example, Eqs. (a) and (c) yield 


/"(x) 


/(x + 2 h) - 2/(x + h) + /(x) 


+ om 


(5.7) 


The third and fourth derivatives can be derived in a similar fashion. The results are 
shown in Tables 5.2a and 5.2b. 



f(x) 

/(x + h) 

f{x + 2h) 

/(x + 3h) 

/(x + Ah) 

hf' (x) 

-1 

i 




}ff"(x) 

1 

-2 

1 



yf f" (x) 

-1 

3 

-3 

1 


/f>/ l4) (x) 

1 

-4 

6 

-4 

1 


Table 5.2a. Coefficients of forward finite difference approximations 
of 0 {h) 



/(x — 4/l) 

/(x -3 h) 

/(x — 2/t) 

/(x - h) 

/M 

hf' (x) 




-1 

1 

iff'{x) 



1 

-2 

1 

Jf f" (x) 


-1 

3 

-3 

1 

/i* / (4) (x) 

1 

-4 

6 

-4 

1 


Table 5.2b. Coefficients of backward finite difference approxima¬ 
tions of 0 [h) 
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5.2 Finite Difference Approximations 


Second Noncentral Finite Difference Approximations 

Finite difference approximations of O(h) are not popular due to reasons that will be 
explained shortly. The common practice is to use expressions of To obtain 

noncentral difference formulas of this order, we have to retain more terms in the 
Taylor series. As an illustration, we will derive the expression for f'(x). We start with 
Eqs. (a) and (c), which are 

/)2 /A lA 

f{x + h) = f(x ) + hf(x) + — f"(x) + - f"(x) + / 4 > (X) + • • • 

2 b 24 

4\? 2h 4 

/(x + 2 h)= /(x) + 2 hf{x) + 2/z 2 f" (x) + — /"'(x) + — / (4) (x) H- 

We eliminate /"(x) by multiplying the first equation by 4 and subtracting it from the 
second equation. The resuit is 

2 h 3 

/(x + 2 h) - 4/(x + h) = —3/(x) - 2 hf'[x) + — /"'(x) H- 

Therefore, 


/M 


-/(x + 2 / 2 ) + 4 /(x+ h) -3/(x) | h 2 ^ ^ i 

0 7, + o / W + 


or 

f W , -/(x + 2»M-4/(x+W- 3/W + 0(tf) ,5. 8) 

Equation (5.8) is called the second forwardfinite difference approximation. 

Derivation of finite difference approximations for higher derivatives involve ad- 
ditional Taylor series. Thus the forward difference approximation for /"(x) utilizes 
series for /(x+ h), f(x + 2 / 7 ) and /(x + 3 h) ; the approximation for /'"(x) involves 
Taylorexpansionsfor /(x+ h), /(x + 2/?), f[x + 3h) and /(x + 4/i),etc. Asyoucan see, 
the computations for high-order derivatives can become rather tedious. The results 
for both the forward and backward finite differences are summarized in Tables 5.3a 
and 5.3b. 



fw 

/(x + h) 

f(x + 2h) 

f[x + 3h) 

/(x + 4 h) 

/(x + 5/z) 

2 hf[x) 

-3 

4 

-1 




ff /"(x) 

2 

-5 

4 

-1 



2 fff"'{x) 

-5 

18 

-24 

14 

-3 


tff 4 \x) 

3 

-14 

26 

-24 

11 

-2 


Table 5.3a. Coefficients of forward finite difference approximations of 0{ff) 
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/(x -5 h) 

/(x — 4/z) 

/(x -3 h) 

/(x - 2 h) 

/(x - h) 

/M 

2 h/'(x) 




1 

-4 

3 

I?f"(x) 



-1 

4 

-5 

2 

2 h 3 f'"(x) 


3 

-14 

24 

-18 

5 

ft*/ (4) (x) 

-2 

11 

-24 

26 

-14 

3 


Table 5.3b. Coefficients of backward finite difference approximations of 0{l£) 


Errors in Finite Difference Approximations 

Observe that in all finite difference expressions the sum of the coefficients is zero. 
The effect on the roundofferror can be profound. If his very small, the values of /(x ), 
/(x ± /i), J'(x ± 2/z) etc. will be approximately equal. When they are multiplied by the 
coefficients and added, several significant figures can be lost. On the other hand, we 
cannot make h too large, because then the truncation error would become excessive. 
This unfortunate situation has no remedy, but we can obtain some relief by taking 
the following precautions: 

• Use double-precision arithmetic. 

• Employ finite difference formulas that are accurate to at least CHh 2 ). 

To illustrate the errors, let us compute the second derivative of /(x) = e~ x at x = 1 
from the Central difference formula, Eq. (5.2). We carry out the calculations with six- 
and eight-digit precision, using different values of h. The results, shown in Table 5.4, 
shouldbe comparedwith /"(1) = e -1 = 0.367 87944. 


h 

6 -digit precision 

8 -digit precision 

0.64 

0.380 610 

0.380 60911 

0.32 

0.371035 

0.37102939 

0.16 

0.368 711 

0.368 664 84 

0.08 

0.368281 

0.368 076 56 

0.04 

0.368 75 

0.367 83125 

0.02 

0.37 

0.3679 

0.01 

0.38 

0.3679 

0.005 

0.40 

0.3676 

0.0025 

0.48 

0.3680 

0.00125 

1.28 

0.3712 


Table 5.4. (e x )" at x = 1 from Central finite dif¬ 
ference approximation 
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5.3 Richardson Extrapolation 


In the six-digit computations, the optimal value of h is 0.08, yielding a resuit 
accurate to three significant figures. Hence three significant figures are lost due to 
a combination of truncation and roundoff errors. Above optimal h, the dominant 
error is due to truncation; below it, the roundoff error becomes pronounced. The 
best resuit obtained with the eight-digit computation is accurate to four significant 
figures. Because the extra precision decreases the roundoff error, the optimal h is 
smaller (about 0.02) than in the six-figure calculations. 


5.3 Richardson Extrapolation 

Richardson extrapolation is a simple method for boosting the accuracy of certain 
numerical procedures, including finite difference approximations (we also use it later 
in other applications). 

Suppose that we have an approximate means of computing some quantity G. 
Moreover, assume that the resuit depends on a parameter h. Denoting the approxi- 
mationbyg(h), wehave G = g( h) + E[h), where E(h) represents the error. Richardson 
extrapolation can remove the error, provided that it has the form E[h ) = ch p , c and p 
being constants. We start by computing g{h) with some value of h, say h= h\. In that 
case we have 


G = gOh) + c/if 


(i) 


Then we repeat the calculation with h = h 2 . so that 

G = g(/ 12 ) + ch% 


(j) 


Eliminating c and solving for G, we obtain from Eqs. (i) and (j) 

G _ Oh/h 2 ) p g{h 2 ) - g{hi) 
[h/hV-l 


(5.9a) 


which is the Richardson extrapolation formula. It is common practice to use h 2 = h\ /2, 
in which case Eq. (5.9a) becomes 


2Pg(fti/2) - gjhf) 
2 p - 1 


(5.9b) 


Let us illustrate Richardson extrapolation by applying it to the finite differ¬ 
ence approximation of (e~ x )" at x= 1. We work with six-digit precision and uti- 
lize the results in Table 5.4. Since the extrapolation works only on the truncation 
error, we must confine h to values that produce negligible roundoff. In Table 5.4 
we have 


g(0.64) = 0.380 610 g(0.32) = 0.371 035 
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The truncation error in the Central difference approximation is H(h) = CK/z 2 ) = + /z 2 + 

c 2 lf + c-$\$ -\ -Therefore, we can eliminate the first (dominant) error term if we 

substitute p = 2 and h\ = 0.64 in Eq. (5.9b). The resuit is 


G = 


2 2 g(0.32) - g(0.64) 
2 2 - 1 


4(0.371035) - 0.380 610 
3 


0.367 84 3 


which is an approximation of (e x )" with the error O ( H '). Note that it is as accurate as 
the best resuit obtained with eight-digit computations in Table 5.4. 

EXAMPLE 5.1 

Given the evenly spaced data points 


X 

0 

0.1 

0.2 

0.3 

0.4 

f(x) 

0.0000 

0.0819 

0.1341 

0.1646 

0.1797 


compute /'(x) and /"(x) at x = 0 and 0.2 using finite difference approximations of 
OGz 2 ). 


Solution We will use linite difference approximations of 0(/z 2 ). From the forward 
difference tables in Table 5.3a we get 


//(0) = —3/(0) + 4/(0.1) — /(0.2) = -3(0)+ 4(0.0819)-0.1341 = Q ^ 


/"( 0 ) = 


2 ( 0 . 1 ) 0.2 
2/(0) - 5/(0.1) + 4/(0.2) - /(0.3) 
(0.1) 2 

2(0) - 5(0.0819) + 4(0.1341) - 0.1646 
(0.1) 2 


= -3.77 


The Central difference approximations in Table 5.1 yield 


/( 0 . 2 ) 


-/(0.1)+ /(0.3) 
2 ( 0 . 1 ) 


-0.0819 + 0.1646 
02 


0.4135 


/'( 0 . 2 ) 


/(0.1)-2/(0.2) + /(0.3) 
(0.1) 2 


0.0819 - 2(0.1341) + 0.1646 

(0Tlj2 


-2.17 


EXAMPLE 5.2 

Use the data in Example 5.1 to compute /(0) as accurately as you can. 

Solution One solution is to apply Richardson extrapolation to finite difference ap¬ 
proximations. We start with two forward difference approximations of O(I^) for /(0): 
one using h = 0.2 and the other one using h = 0.1. Referring to the formulas of O(/z 2 ) 
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5.3 Richardson Extrapolation 


in Table 5.3a, we get 


g(0.2) 
g(0.1) = 


-3/(0)+ 4/(0.2)- /(0.4) 
2 ( 0 . 2 ) 

3/(0)+4/(0.1)- /(0.2) 
2 ( 0 . 1 ) 


3(0) +4(0.1341) - 0.1797 
(14 

-3(0)+ 4(0.0819)- 0.1341 
02 


0.8918 

= 0.9675 


Recall that the error in both approximations is of the form E(h ) = C| /t 2 + c 2 IP + 

C3/2 6 H-. We can now use Richardson extrapolation, Eq. (5.9), to eliminate the dom- 

inant error term. With p= 2 we obtain 

/'(0) * G = 22g(0 - 1 ? ) - gt °- 2) = 4(0-9675)- 0.8918 = Q ^ 

2 2 — 1 3 

which is a finite difference approximation of O (/7 1 ). 

EXAMPLE 5.3 



The linkage shown has the dimensions a = 100 mm, b= 120 mm, c = 150 mm 
and d= 180 mm. It can be shown by geometry that the relationship between the 
angles a and p is 

(d— acos a — b cos fi) 2 + (asina + bsin/i ) 2 — c 2 = 0 

For a given value of a, we can solve this transcendental equation for /i by one of the 
root-finding methods in Chapter 4. This was done with a = 0°, 5°, 10°,..., 30°, the 
results being 


a (deg) 

0 

5 

10 

15 

20 

25 

30 

p (rad) 

1.6595 

1.5434 

1.4186 

1.2925 

1.1712 

1.0585 

0.9561 


If link AB rotates with the constant angular velocity of 25 rad/s, use finite difference 
approximations of Otii 2 ) to tabulate the angular velocily dfi/dt. of link BC againsta. 


Solution The angular speed of BC is 


dp 

dt 


dp da dp 

= 25-/- rad/s 
da dt da 
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where dfi/da can be computed from finite difference approximations using the data 
in the table. Forward and backward differences of (!) (/t 2 ) are used at the endpoints, 
Central differences elsewhere. Note that the increment of a is 


h = 


The computations yield 


(5deg) ^^^rad/deg^ = 0.087 266 rad 


„ —3/3(0°) + 4/1(5°) — /6(10°) -3(1.6595) +4(1.5434) - 1.4186 

p(U J = 25-—- = 25- 


2 h 


2(0.087 266) 


= —32.01 rad/s 


*(5») = 25 ^ 10O l^ ( ° O) = 25 '•; ll86 „,'; 6595 = -34.51 rad/s 


2 h 


2(0.087 266) 


etc. 


The complete set of results is 


a (deg) 

0 

5 

10 

15 

20 

25 

30 

P (rad/s) 

-32.01 

-34.51 

-35.94 

-35.44 

-33.52 

-30.81 

-27.86 


5.4 Derivatives by Interpolation 

If /(x) is given as a set of discrete data points, interpolation can be a very effective 
means of computing its derivatives. The idea is to approximate the derivative of /(x) 
by the derivative of the interpolant. This method is particularly useful if the data points 
are located at uneven intervals of x, when the finite difference approximations listed 
in the last article are not applicable. 12 

Polynomial Interpolant 

The idea here is simple: fit the polynomial of degree n 

P n -\{x) = flo + <hx + (hx 2 -\ -h a n x n 

through n+ 1 data points and then evaluate its derivatives at the given x. As pointed 
out in Section 3.2, it is generally advisable to limit the degree of the polynomial to 
less than six in order to avoid spurious oscillations of the interpolant. Since these 
oscillations are magnified with each differentiation, their effect can devastating. In 

12 It is possible to derive finite difference approximations for unevenly spaced data, but they would 
not be as accurate as the formulas derived in Section 5.2. 
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5.4 Derivatives by Interpolation 


view of the above limitation, the interpolation is usually a local one, involving no more 
than a few nearest-neighbor data points. 

For evenly spaced data points, polynomial interpolation and finite difference 
approximations produce identical results. In fact, the finite difference formulas are 
equivalent to polynomial interpolation. 

Several methods of polynomial interpolation were introduced in Section 3.2. Un- 
fortunately, none of them is suited for the computation of derivatives of the inter¬ 
polant. The method that we need is one that determines the coefficients ao, «i,..., a n 
of the polynomial. There is only one such method discussed in Chapter 3: the least- 
squaresflt. Although this method is designed mainly for smoothing of data, it will 
carry out interpolation if we use m= n in Eq. (3.22)—recall that mis the degree of the 
interpolating polynomial and n+ 1 represents the number of data points to be fitted. 
If the data contains noise, then the least-squares fit should be used in the smoothing 
mode, that is, with m< n After the coefficients of the polynomial have been found, 
the polynomial and its first two derivatives can be evaluated efficiently by the function 
evalPoly listed in Section 4.7. 


Cubic Spline Interpolant 


Due to its stiffness, cubic spline is a good global interpolant; moreover, it is easy to 
differentiate. The first step is to determine the second derivatives fc,- of the spline at 
the knots by solving Eqs. (3.11). This can be done with the function curvature s in 
the module cubicSpline listed in Section 3.3. The first and second derivatives are 
then computed from 


fu +iM 


h 

6 


3(x — x,- + i) 2 

Xi - X i+ i 


- (x,- - Xi+i) 


k i+ 1 f 3(x-x,) 2 

6 [_ Xi- Xj+i 


(X; - Xi+i) 


+ yt - yt+i 

Xi - X i+ 1 


(5.10) 


f" 

J 1,1+1 


(x) 


X-Xf+1 , X-Xj 

k - kj + \ ——- 

Xi Xi+i Xi Xi+i 


which are obtained by differentiation of Eq. (3.10). 


EXAMPLE 5.4 

Given the data 


(5.11) 


X 

1.5 

1.9 

2.1 

2.4 

2.6 

3.1 

f(x) 

1.0628 

1.3961 

1.5432 

1.7349 

1.8423 

2.0397 
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compute /'(2) and /"(2) using (1) polynomial interpolation over three nearest- 
neighbor points, and (2) natural cubic spline interpolant spanning all the data 
points. 

Solution of Part (1) The interpolant is P 2 (x) = flo + fliX + a 2 x 2 passing through the 
points at x = 1.9, 2.1 and 2.4. The normal equations, Eqs. (3.23), of the least-squares 
fit are 


n £x; £*? 


flo 


1 

w 

1_ 

£*f £*f 


CL\ 

= 

£ yM 

X x f E 4. 


CI 2 


.£ ytxf_ 


After substituting the data, we get 


3 6.4 13.78 ‘ 


flb 


’ 4.6742 “ 

6.4 13.78 29.944 


G\ 

= 

10.0571 

13.78 29.944 65.6578 


Cti 


21.8385 


whichyields a = [ —0.7714 1.5075 -0.1930] 7 . 

The derivatives of the interpolant are P^ix) = a\ +2a 2 x and P^ix) = 2Ag¬ 
ili e re fore, 

/'(2) « P 2 {2) = 1.5075 + 2(—0.1930X2) = 0.7355 
/"(2) « P 2 [2) = 2(—0.1930) = -0.3860 

Solution of Part (2) We must first determine the second derivatives lq of the spline 
at its knots, after which the derivatives of /(x) can be computed from Eqs. (5.10) and 
(5.11). The first part can be carried out by the following small program: 

#!/usr/bin/python 
## example5_4 

from cubicSpline import curvatures 
from LUdecomp3 import * 
from numarray import array 

xData = array([1.5, 1.9, 2.1, 2.4, 2.6, 3.1]) 

yData = array([1.0628, 1.3961, 1.5432, 1.7349, 1.8423, 2.0397]) 
print curvatures(LUdecomp3,LUsolve3,xData,yData) 
raw_input(’’Press return to exit’’) 

The output of the program, consisting of ko to fe, is 

[ 0. -0.4258431 -0.37744139 -0.38796663 -0.55400477 0. ] 


Press return to exit 
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Since x = 2 lies between knots 1 and 2, we must use Eqs. (5.10) and (5.11) with 
i = 1. Thisyields 


/( 2 ) » / i , 2 ( 2 ) = -j 


3(x- x 2 ) 2 


h 

6 


Xi -x 2 
3(x - Xi) 2 


Xi - x 2 


- (Xi - x 2 ) 


- (Xi - x 2 ) 

yi - y2 


+ 


Xi - x 2 


(-0.4258) 


(-0.3774) 


3(2-2.1) 2 

(- 0 . 2 ) 

3(2 - 1.9) 2 


(- 0 . 2 ) 


- (- 0 . 2 ) 

- (- 0 . 2 ) 


+ 


1.3961 - 1.5432 


(- 0 . 2 ) 


= 0.7351 


/"(2) « fi 2 (2) = h ^ - k 2 X Xl 


Xi - x 2 


Xi - x 2 


2—21 2—19 

= (-0.4258) - ^ - (-0.3774) , „ 1 = -0.4016 


(- 0 . 2 ) 


(- 0 . 2 ) 


Note that the Solutions for f (2) in parts (1) and (2) differ only in the fourth significant 
figure, but the values of /"(2) are much farther apart. This is not unexpected, consid- 
ering the general rule: the higher the order of the derivative, the lower the precision 
with which it can be computed. It is impossible to teli which of the two results is 
better without knowing the expression for /(x). In this particular problem, the data 
points fall on the curve /(x) = x 2 e~ x/2 , so that the “true” values of the derivatives are 
/'(2) = 0.7358 and /"(2) = -0.3679. 

EXAMPLE 5.5 

Determine /'(0) and /'(1) from the following noisy data 


X 

0 

0.2 

0.4 

0.6 

/M 

1.9934 

2.1465 

2.2129 

2.1790 

X 

0.8 

1.0 

1.2 

1.4 

/M 

2.0683 

1.9448 

1.7655 

1.5891 


Solution We used the program listed in Example 3.10 to find the best polynomial fit 
(in the least-squares sense) to the data. The program was run three times with the 
following results: 

Degree of polynomial ==> 2 
Coefficients are: 

[2.0261875 0.64703869 -0.70239583] 


































194 


Numerical Differentiation 


Std. deviation = 0.0360968935809 
Degree of polynomial ==> 3 
Coefficients are: 

[1.99215 1.09276786 -1.55333333 0.40520833] 

Std. deviation = 0.0082604082973 

Degree of polynomial ==> 4 
Coefficients are: 

[1.99185568 1.10282373 -1.59056108 0.44812973 -0.01532907] 

Std. deviation = 0.00951925073521 

Degree of polynomial ==> 

Finished. Press return to exit 

Based on Standard deviation, the cubic seems to be the best candidate for the 
interpolant. Before accepting the resuit, we compare the plots of the data points and 
the interpolant—see the figure. The fit does appear to be satisfactory 



Approximating j\x) by the interpolant, we have 

/( x) « flo + ai* + ctzx 2 + fyx 3 


so that 


/'(x) w Ol + 2a 2 x + 3asx 2 


The re fore 


/'(0) « ai = 1.093 

/(1) = ai +20.2 + 303 = 1.093 + 2(-1.553) +3(0.405) = -0.798 
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5.4 Derivatives by Interpolation 


In general, derivatives obtained from noisy data are at best rough approximations. 

In this problem, the data represent /(x) = (x + 2)/ cosh x with added random noise. 

Thus /'(x) = [l - (x + 2) tanhx] / coshx, so that the “correct” derivatives are /'(0) = 

1.000 and /'(1) = -0.833. 

PROBLEM SET 5.1 

1. Given the values of /(x) at the points x, x — h\ and x + h 2 , where hi / h 2 , de- 
termine the linite difference approximation for /"(x). What is the order of the 
truncation error? 

2. Given the first backward finite difference approximations for /'(x) and /"(x), 
derive the first backward finite difference approximation for /'"(x) using the 
operation /'"(x) = [/"(x)]\ 

3. Derive the Central difference approximation for /"(x) accurate to 0{lf) by apply- 
ing Richardson extrapolation to the Central difference approximation of 0{}fi). 

4. Derive the second forward finite difference approximation for /"'(x) from the 
Taylor series. 

5. Derive the first Central difference approximation for / (4) (x) from the Taylor series. 

6. Use finite difference approximations of O0 i 2 ) to compute /'(2.36) and /"(2.36) 
from the data 


X 

2.36 

2.37 

2.38 

2.39 

f(x) 

0.85866 

0.86289 

0.86710 

0.87129 


7. Estimate /'(1) and /"(1) from the following data: 


X 

0.97 

1.00 

1.05 

/M 

0.85040 

0.84147 

0.82612 


8. Given the data 


X 

0.84 

0.92 

1.00 

1.08 

1.16 

/M 

0.431711 

0.398519 

0.367879 

0.339596 

0.313486 


calculate /"(1) as accurately as you can. 
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9. Use the data in the table to compute /'(0.2) as accurately as possible. 


X 

0 

0.1 

0.2 

0.3 

0.4 

/M 

0.000 000 

0.078348 

0.138910 

0.192916 

0.244 981 


10. Using five significant figures in the computations, determine dfsin x)/dx at 
x = 0.8 from (a) the first forward difference approximation and (b) the first Central 
difference approximation. fn each case, use h that gives the most accurate resuit 
(this requires experimentation). 

11. ■ Use polynomial interpolation to compute f' and f" at x = 0, using the data 


X 

-2.2 

-0.3 

0.8 

1.9 

/M 

15.180 

10.962 

1.920 

-2.040 


12. ■ 



The crank AB of length R = 90 mm is rotating at the constant angular speed of 
dd/dt = 5000 rev/min. The position of the piston C can be shown to vary with the 
angle 9 as 

x = R ^cos 8 + \j 2.5 2 — sin 2 8^j 

Write a program that computes the accelerationofthe piston at 6 = 0°,5°, 10°,..., 
180° by numerical differentiation. 
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5.4 Derivatives by Interpolation 


The radar stations A and B, separated by the distance a = 500 m, track the plane 
C by recording the angles a and fi at one-second intervals. If three successive 
readings are 


t (s) 

9 

10 

11 

a 

54.80° 

54.06° 

53.34° 

p 

65.59° 

64.59° 

63.62° 


calculate the speed v of the plane and the climb angle y at t = 10 s. The coordinates 
of the plane can be shown to be 

tand Lan a tan fi 

x = a - y = a - 

tan/l — tana Lan fi — lan a 



Geometric analysis of the linkage shown resulted in the following table relating 
the angles 8 and fi: 


9 (deg) 

0 

30 

60 

90 

120 

150 

P (deg) 

59.96 

56.42 

44.10 

25.72 

-0.27 

-34.29 


Assuming that member AB of the linkage rotates with the constant angular ve- 
locity d8/dt= 1 rad/s, compute dfi/dt in rad/s at the tabulated values of 9. Use 
cubic spline interpolation. 
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Numerical Integration 


Compute / fl b f(x) dx, where /(x) is a given function 


6.1 


Introduction 

Numerical integration, also known as quadrature, is intrinsically a much more accu¬ 
rate procedure than numerical differentiation. Quadrature approximates the definite 
integral 





by the sum 


n 


i = J2 Ai /ta) 


where the nodal abscissas x, and weights Ai depend on the particular rule used for the 
quadrature. All rules of quadrature are derived from polynomial interpolation of the 
integrand. Therefore, they work best if /(x) can be approximated by a polynomial. 

Methods of numerical integration can be divided into two groups: Newton-Cotes 
formulas and Gaussian quadrature. Newton-Cotes formulas are characterized by 
equally spaced abscissas, and include well-known methods such as the trapezoidal 
rule and Simpson’s rule. They are most useful if /(x) has already been computed at 
equal intervals, or can be computed at low cost. Since Newton-Cotes formulas are 
based on local interpolation, they require only a piecewise fit to a polynomial. 

In Gaussian quadrature the locations of the abscissas are chosen to yield the best 
possible accuracy. Because Gaussian quadrature requires fewer evaluations of the in¬ 
tegrand for a given level of precision, it is popular in cases where /(x) is expensive 


6.1 
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6.2 Newton-Cotes Formulas 


to evaluate. Another advantage of Gaussian quadrature is its ability to handle inte- 
grable singularities, enabling us to evaluate expressions such as 


f 


gW 

n/ 1 — Jk 


.dx 


provided that g(x) is a well-behaved function. 


6.2 Newton-Cotes Formulas 



Figure 6.1. Polynomial approximation of f{x). 


Consider the definite integral 


/' 

J a 


f(x ) dx 


( 6 . 1 ) 


We divide the range of integration [a, b) into n equal intervals of length h= [b — a)/n, 
as shown in Fig. 6.1, and denote the abscissas of the resulting nodes by x 0 . X|,..., x n . 
Next we approximate /(x) by a polynomial of degree n that intersects all the nodes. 
Lagrange’s form of this polynomial, Eq. (3.1a), is 

n 

Pnlx) = f(M)(-i(x) 

1=0 

where l\(x) are the Cardinal functions defined in Eq. (3.1b). Therefore, an approxima¬ 
tion to the integral in Eq. (6.1) is 


/ = 


f P n {x)dx=Y^ /U;) f ti[x)dx = Y] Aif(Xi) 
J a i=0 L Ja J i=0 


(6.2a) 


where 


f b 

Ai = / ii[x)dx, i = 0, 1, ...,n 


(6.2b) 


Equations (6.2) are the Newton-Cotes formulas. Classical examples of these formu¬ 
las are the trapezoidal rule (n= 1), Simpson’s rule (n = 2) and Simpson’s 3/8 rule 
{n = 3). The most important of these is the trapezoidal rule. It can be combined with 
Richardson extrapolation into an efficient algorithm known as Romberg integration, 
which makes the other classical rules somewhat redundant. 
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Trapezoidal Rule 



Figure 6.2. Trapezoidal rule. 


If n = 1 (one panel), as illustrated in Fig. 6.2, we have i 0 = (x — x\ )/(x 0 — Xi) = 
— (x — b) / h. Therefore, 

*= (*-»)d*=Yh ib - af =l 

Also l\ = (x — Xo)/(xi — Xo) = (x - a)/h, so that 

A '-li 


Substitution in Eq. (6.2a) yields 

I = (/(«) + /(&)] | 


(6.3) 


which is known as the trapezoidal rule. It represents the area of the trapezoid in Fig. 6.2. 
The error in the trapezoidal rule 


E = 



- I 


is the area of the region between /(x) and the straight-line interpolant, as indicated 
in Fig. 6.2. It can be obtained by integrating the interpolation error in Eq. (3.3): 


E = 


1 . [ (x-Xo)(x-Xi)/"(£)dx= [ (x — a)[x— b)dx 

Ja ^ Ja 


= -l(b-a) 3 /'(f) = -^/"(§) 


(6.4) 


Composite Trapezoidal Rule 
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In practice the trapezoidal rule is applied in a piecewise fashion. Figure 6.3 shows 
the region ( a, b) divided into n panels, each of width h. The function /( x) to be inte- 
grated is approximated by a straight line in each panel. From the trapezoidal rule we 
obtain for the approximate area of a typical (zth) panel 

Ii = \ fixi) + f(xt+ 1 )] ^ 

Flence total area, representing f{x) dx, is 

/ = £// = [ftxo) + 2/te) +2 f{x 2 ) + ■■■ + 2/(x„_i) + f(x n )} | (6.5) 

1=0 z 

which is the composite trapezoidal rule. 

The truncation error in the area of a panel is, from Eq. (6.4), 

/f 3 

Ei =~nf ''«') 

where flies in (jc,, jc,- + i). Flence the truncation error in Eq. (6.5) is 

n -1 iX n— 1 

i =0 i =0 

But 

E = «f 

i=0 


where /" is the arithmetic mean of the second derivatives. If f"[x) is continuous, there 
must be a point £ in ( a , b) at which /"(£) = /", enabling us to write 

E = «ni) = ^/"(?) 

!=0 77 


Therefore, Eq. (a) becomes 


£ 


(h — a ) /z 2 
12 


/"(?) 


( 6 . 6 ) 


It would be incorrect to conclude from Eq. (6.6) that E = c/z 2 (c being a constant), 
because /"(§) is not entirely independent of h. A deeper analysis of the error 13 shows 
that if /(x) and its derivatives are finite in {a, b), then 

E = C\lf -\- C 2 lt^ C 3 /z 13 ) ■ ■ ■ (6.7) 


13 The analysis requires familiarity with the Euler-Maclaurin summatiori formula, which is covered 
in advanced texts. 
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Numerical Integration 


Recursive Trapezoidal Rule 

Let 4 be the integral evaluated with the composite trapezoidal rule using 2 fc_1 panels. 
Note that if k is increased by one, the number of panels is doubled. Using the notation 

H = b— a 

we obtain from Eq. (6.5) the following results for k = 1,2 and 3. 


/c= 1 (1 panel): 

h = Ifta) + fm | 

k= 2 (2 panels): 


h = 


f{a) + 2 f [ a+ — ) + f{b) 


H 1 / H\ H 

— — —4 T f [ a-\- — ) — 
4 2 1 J \ 2 2 


( 6 . 8 ) 


k= 3 (4 panels): 


h = 


H 


/(rz) + 2 /1 cz H —— \ + 2fl a -\—— ) + 2 /1 rz + 


H 


3 H 


+ m 


H 

¥ 


— —12 
2 


H 


f\ 5T ) + / ( a_l - 7~ 


3 H 


H 

~4 


We can now see that for arbitrary fc > 1 we have 


ofc—2 

r 1 H 

4 - yh-i + 2 ¥ J 


ci -\- 


(2i — l)H 
2 k ~ x 


k= 2,3,... 


(6.9a) 


which is the recursive trapezoidal rule. Observe that the summation contains only the 
new nodes that were created when the number of panels was doubled. Therefore, 
the computation of the sequence I\, I 2 , /3,..., 4 from Eqs. (6.8) and (6.9) involves 
the same amount of algebra as the calculation of 4 directly from Eq. (6.5). The advan- 
tage of using the recursive trapezoidal rule is that it allows us to monitor convergence 
and terminate the process when the difference between 4-i and 4 becomes suffi- 
ciently small. A form of Eq. (6.9a) that is easier to remember is 

/(/z) = i/(2/z) + /2^/U new ) (6.9b) 

where h= H/n is the width of each panel. 


■ trapezoid 

The function trapezoid computes 4 (inew), given I^i (Iold) from Eqs. (6.8) and 
(6.9). We can compute / f(x) dx by calling trapezoid with Ic = 1, 2, ... until the 
desired precision is attained. 
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6.2 Newton-Cotes Formulas 


## module trapezoid 

’’’ Inew = trapezoid(f,a,b,Iold,k). 

Recursive trapezoidal rule: 

Iold = Integral of f(x) from x = a to b computed by 
trapezoidal rule with 2~(k-l) panels. 

Inew = Same integral computed with 2"k panels. 


def trapezoid(f,a,b,Iold,k): 

if k == 1:Inew = (f(a) + f(b))*(b - a)/2.0 
else : 

n=2**(k-2) # Number of new points 

h = (b - a)/n # Spacing of new points 

x = a + h/2.0 # Coord. of lst new point 

sum = 0.0 

for i in range(n): 

sum = sum + f(x) 
x = x + h 

Inew = (Iold + h*sum)/2.0 
return Inew 

Simpson's Rules 



Figure6.4. Simpson’s 1/3 rule. 


Simpsoris 1/3 rule can be obtained from Newton-Cotes formulas with n = 2; that 
is, by passing a parabolic interpolant through three adjacent nodes, as shown in 
Fig. 6.4. The area under the parabola, whichrepresents an approximation of f[x) dx, 
is (see derivation in Example 6.1) 


I = 


f{a) + 4/ 


a+ b 


+ m 


(a) 
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Numerical Integration 


To obtain the composite Simpsoris 1/3 rule, the integration range ( a, b) is divided into 
npanels (neven) ofwidth h= ( b— a)/ne ach, asindicatedinFig. 6.5. ApplyingEq. (a) 
to two adjacent panels, we have 


L 


Xi +2 h 

/M dx « \f(Xi) + 4 f[x i+ 1 ) + fUi+2)] - 


(b) 


Substituting Eq. (b) into 


rb rx m n r „x i+ 2 

/ J'(x) dx = / f(x) dx = '^2 / f{x)dx 

Jct Jx o i=0,2,... \-Jx t 


yields 



I — [/(-'3o) + 4/te) + 2/te) + 4/te) + • • • 
• • • + 2f{x n - 2 ) + 4/(x,j_i) + f(x n )] — 


( 6 . 10 ) 


The composite Simpson’s 1/3 rule in Eq. (6.10) is perhaps the best-known method of 
numerical integration. Its reputation is somewhat undeserved, since the trapezoidal 
rule is more robust, and Romberg integration is more efficient. 

The error in the composite Simpson's rule is 


E = 


[b — a) Ift 

180 


/ (4, (§) 


( 6 . 11 ) 


from which we conclude that Eq. (6.10) is exact if f(x) is a polynomial of degree three 
or less. 

Simpson’s 1/3 rule requires the number of panels n to be even. If this condition 
is not satisfied, we can integrate over the first (or last) three panels with Simpsoris 
3/8 rule : 


I = 


lf(x 0 ) + 3/(xi) +3/(x 2 ) + /te)] 


3 h 
~ 8 ~ 


( 6 . 12 ) 


and use Simpson’s 1/3 rule for the remaining panels. The error in Eq. (6.12) is of the 
same order as in Eq. (6.10). 


EXAMPLE 6.1 

Derive Simpson’s 1/3 rule from Newton-Cotes formulas. 


Solution Referring to Fig. 6.4, we see that Simpson’s 1/3 rule uses three nodes located 
at Xq = a, Xi = (a+ b) /2 and x 2 = b. The spacing of the nodes is h= [b — a)/ 2. The 
Cardinal functions of Lagrange’s three-point interpolation are (see Section 3.2) 

(x X\)(x x 2 ) {x- XoKx- x 2 ) 

fote =- fiM =- 

te - JCi) te - X 2 ) te - Xq) te - X 2 ) 




{X- Xq)[X- Xi) 
(X2 - Xo)te - Xi) 
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6.2 Newton-Cotes Formulas 


The integration of these functions is easier if we introduce the variable £ with origin 
at X\. Then the coordinates of the nodes are § 0 = — h, f j = 0, £ 2 = h and Eq. (6.2b) 
becomes At = f'’li(x)dx = f^ h t ,■ (£) cg. Therefore, 


Aq = 


Ai = 


Ai = 



(g - 0) (g - /z) 

(-h)(-2h) 


(g + /2) (g - 
(h){-h) 



cg + /dcg — o) 

(2h)(h) 




i r h , , J h 

1 [ h ? ,, , 4/2 

-wJJ 

1 f h o , 1 h 

2 WjJ* + ^~3 


Equation (6.2a) then yields 


I = 


2 


Ea-/w 

i=0 


/(«) + 4 /(~y“) + ^ 


h 

3 


which is Simpsoris 1/3 rule. 


EXAMPLE 6.2 

Evaluate the bounds on sin(x) dx with the composite trapezoidal rule using (1) 
eight panels and (2) sixteen panels. 

Solution of Part (1) With 8 panels there are 9 nodes spacedat h= n/8. The abscissas 
of the nodes are x,- = ii r/8, i = 0,1,..., 8. From Eq. (6.5) we get 


/ = 


E lTT 

sin-h 

R 


sm 


sm7r 


j=i 


— = 1.97423 
16 


The error is given by Eq. (6.6): 

[b— a) h 2 


E = —- 


f"& = ~ 


(7r — 01(71/8)“, . ,, jt 


(— sini?) =-sin? 

768 


12 J ’ 12 
where 0 < § < ;r. Since we do not know the value of g we cannot evaluate E, but we 
can determine its bounds: 


Emin = — sin(0) = 0 
768 


£ max = —— sin — = 0.04037 
768 2 


Therefore, I + £ m i n < f* sin(x) dx < I + E max , or 


1.97423 </ sin(x) dx < 2.014 60 


/- 


The exact integral is, of course, 2. 

Solution of Part (2) The new nodes created by the doubling of panels are located at 
midpoints of the old panels. Their abscissas are 

Xj = jr/16 + jit /8 = (1 + 2 j)jr/16, j = 0,1,..., 7 
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Numerical Integration 


Using the recursive trapezoidal rule in Eq. (6.9b), we get 


1.974 23 7T ^ 

I =-1-> sin 

2 16 fe 


(1+2 j)ii 
16 


= 1.993 58 


and the bounds on the error become (note that E is quartered when h is halved) 
E min = 0, E max = 0.040 37/4 = 0.010 09. Hence 


1.99358 < f sin(x) dx < 2.00367 

J o 


EXAMPLE 6.3 

Estimate / Q 2 5 /(x) dx from the data 


X 

0 

0.5 

1.0 

1.5 

2.0 

2.5 

/M 

1.5000 

2.0000 

2.0000 

1.6364 

1.2500 

0.9565 


Solution We will use Simpson's rules, since they are more accurate than the trape¬ 
zoidal rule. Because the number of panels is odd, we compute the integral over the 
first three panels by Simpson’s 3/8 rule, and use the 1/3 rule for the last two panels: 

I = [/(0) + 3/(0.5) + 3/(1.0) + /(1.5)] 

O 

+ [/(1.5)+4/(2.0)+ /(2.5)] 

= 2.8381 + 1.2655 = 4.1036 

EXAMPLE 6.4 

Use the recursive trapezoidal rule to evaluate // Jx cos x dx to six decimal places. 
How many panels are needed to achieve this resuit? 

Solution The programlistedbelowutilizes the function trapezoid. 

#!/usr/bin/python 
## example6_4 

from math import sqrt,cos,pi 
from trapezoid import * 

def f(x): return sqrt(x)*cos(x) 

Iold =0.0 

for k in range(l,21): 

Inew = trapezoid(f,0.0,pi,Iold,k) 

if (k > 1) and (abs(Inew - Iold)) < 1.0e-6: break 
Iold = Inew 
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6.3 Romberg Integration 


print ’’Integral =’’,Inew 
print ’’nPanels =’’,2**(k-l) 
raw_input(’’ \nPress return to exit’’) 

The output from the program is: 

Integral = -0.894831664853 
nPanels = 32768 

Hence */xcosxdx = -0.894 832 requiring 32 768 panels. The slow conver- 
gence is the resuit of all the derivatives of f\x) being singular at x = 0. Consequently, 

the error does not behave as shown in Eq. (6.7): E = Citf + c 2 }ft H-, but is unpre- 

dictable. Difficulties of this nature can often be remedied by a change in variable. In 
this case, we introduce t = Jx, so that dt = dx/(2*/x) = dx/{2t) } o r dx = 2 tdt. Thus 



Evaluation of the integral on the right-hand side was completed with 4096 panels. 

6.3 Romberg Integration 

Romberg integration combines the trapezoidal rule with Richardson extrapolation 
(see Section 5.3). Let us first introduce the notation 


where, as before, /, represents the approximate value of f(x) dx computed by the 


recursive trapezoidal rule using 2‘ 1 panels. Recall that the error in this approximation 
is E = Ci h 2 + c 2 -\ -, where 


is the width of a panel. 

Romberg integration starts with the computation of R\ i = Ii (one panel) and 
f? 2 ,i = h (two panels) from the trapezoidal rule. The leading error term c) /r 2 is then 
eliminated by Richardson extrapolation. Using p = 2 (the exponent in the leading 
error term) in Eq. (5.9) and denoting the resuit by R 2 , 2 » we obtain 



(a) 
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Numerical Integration 


It is convenient to store the results in an array of the form 

^ 2,1 Rz ,2 

The next step is to calculate f? 3 ,i = /3 (four panels) and repeat Richardson extrap- 
olation with J ? 2j 1 and /? 3 1 , storing the resuit as J? 32 : 

^3,2 = 2^3,1 — 3^2,1 (b) 

The elements of array R calculated so far are 


R1.1 

Ri,\ R 2.2 
f?3,i R 3.2 


Both elements of the second column have an error of the form c 2 /t 4 , which can also 
be eliminated with Richardson extrapolation. Using p = 4 in Eq. (5.9), we get 


f?3,3 


2 4 1?3,2 — f?2,2 
2 4 - 1 




(c) 


This resuit has an error of Oifft). The array has now expanded to 


« 1,1 

f?2,l R2,2 

_f?3,l f?3,2 f?3,3 

After another round of calculations we get 


1 

£3 



f?2,l 

R2,2 


R 3 .I 

R 3.2 

i?3,3 

1 

£ 

Ra,2 

Ra,3 R4A 


where the error in R 4 4 is O ( U'). Note that the most accurate estimate of the integral is 
always the last diagonal term of the array. This process is continued until the differ- 
ence between two successive diagonal terms becomes sufficiently small. The general 
extrapolation formula used in this scheme is 


Rij = 


Ri,j~ 1 — Ri-ij-i 

4t _1 - 1 ’ 


i >1, j = 2, 3, 


(6.13a) 


Apictorial representation of Eq. (6.13a) is 
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6.3 Romberg Integration 




\ 


\ 


Ri 


ij -1 


R 


‘•i 


where the multipliers a and f J > depend on j in the following manner: 


j 

2 

3 

4 

5 

6 

a 

- 1/3 

-1/15 

-1/63 

-1/255 

-1/1023 

p 

4/3 

16/15 

64/63 

256/255 

1024/1023 


(6.13b) 


(6.13c) 


The triangular array is convenient for hand computations, but computer imple- 
mentation of the Romberg algorithm can be carried out within a one-dimensional 
array R'. After the first extrapolation—see Eq. (a )—Rn is never used again, so that it 
can be replaced with i? 2 , 2 - As a resuit, we have the array 

~ R\ = R 2 ,2~ 

_R'= R 2 ,i_ 

In the second extrapolation round, defined by Eqs. (b) and (c), J ? 3 , 2 overwrites /i 2 ,i, 
and f ? 3 3 replaces /i 2 2 , so that the array contains 

"*1 = Rs, 3 " 

R 2 = Rs,2 
_R^ = R 3 ,i_ 

and so on. In this manner, R\ always contains the best current resuit. The extrapolation 
formula for the k th round is 

4 fc -JR' . - R!. 

R: = --ttl- L / = Jfc— l, fc— 2.1 (6.14) 

J 4 k-J _ 1 ’ J ’ 


■ romberg 

The algorithm for Romberg integration is implemented in the function romberg. It 
returns the integral and the number of panels used. Richardson’s extrapolation is 
carried out by the subfunction richardson. 

## module romberg 

’’’ I,nPanels = romberg(f,a,b,tol=l.0e-6). 

Romberg integration of f(x) from x = a to b. 

Returns the integral and the number of panels used. 
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Numerical Integration 


from numarray import zeros,Float64 
from trapezoid import * 

def romberg(f,a,b,tol=l.Oe-6) : 

def richardson(r,k): 

for j in range(k-l,0,-1): 
const = 4.0**(k-j) 

r[j] = (const*r[j+1] - r[j])/(const - 1.0) 
return r 

r = zeros((21),type=Float64) 
r[l] = trapezoid(f,a,b,0.0,1) 
r_old = r[l] 
for k in range(2,21): 

r[k] = trapezoid(f,a,b,r[k-1],k) 
r = richardson(r,k) 

if abs(r[1]-r_old) < tol*max(abs(r[1]),1.0): 


return r[1],2**(k-1) 


r_old = r[l] 

print ’’Romberg quadrature did not converge’’ 

EXAMPLE 6.5 

Showthat Rk .2 in Romberg integration is identical to the composite Simpson’s 1 /3 rule 
in Eq. (6.10) with 2 k ~ 1 panels. 

Solution Recall that in Romberg integration /4. i = 4 denoted the approximate inte- 
gral obtained by the composite trapezoidal rule with n = 2 k ~ l panels. Denoting the 
abscissas of the nodes by Xq, X \,..., x,„ we have from the composite trapezoidal rule 
in Eq. (6.5) 


h 

2 


n — i y 

Rk,l = Ik = fiXo) + 2 /ta) + - f(Xn) 


i= 1 


When we halve the number of panels (panel width 2 h), only the even-numbered 
abscissas enter the composite trapezoidal rule, yielding 


n —2 


Rk- i,i = 4-i = fixo ) + 2 Y] f(Xi ) + f{x n ) h 
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6.3 Romberg Integration 


Applying Richardson extrapolation yields 


Rk,2 


4 1 

3 Rk l ~ 3 ' 1,1 


\ /(*o) + \ E f(*i) + 


1=1,3,. 


| E /w + i/uj 


h 


which agrees with Eq. (6.10). 


EXAMPLE 6.6 

Use Romberg integration to evaluate f* f(x) dx, where f{x) = sin x. Work with four 
decimal places. 

Solution From the recursive trapezoidal rule in Eq. (6.9b) we get 

Ri,i = IW = | [/( 0 ) + fW] = 0 

R 2 , i = 10 r/2) = i/(;r) + |/(tt/2) = 1.5708 

R 3 ,i = /(tt/4) = |/(tt/2) + | [/(tt/4) + /(3jt/4)] = 1.8961 

R*,i = Kn/8 ) = 4/(tt/4) + J [/(tt/8) + /(3tt/ 8) + /(5jr/8) + /(7;r/8)] 

Z 8 

= 1.9742 

Using the extrapolation formulas in Eqs. (6.13), we can now construet the following 
table: 


~Ri,i 


0 

R 2 ,l R 2 ,2 


1.5708 2.0944 

f?3,l R3. 2 R3,3 


1.8961 2.0046 1.9986 

_Ri, 1 Ri,2 Rl,3 Raa _ 


1.9742 2.0003 2.0000 2.0000_ 


It appears that the procedure has converged. Therefore, sin x dx = R 4 A = 2.0000, 
which is, of course, the correct resuit. 


EXAMPLE 6.7 

Use Romberg integration to evaluate 2x 2 cos x 2 dx and compare the results with 
Example 6.4. 

Solution 

#!/usr/bin/python 
## example6_7 

from math import cos,sqrt,pi 
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from romberg import * 

def f(x): return 2.0*(x**2)*cos(x**2) 

I,n = rombergCf,0,sqrt(pi)) 
print ’’Integral =’ ’ , I 
print ’'nPanels =’ ’ ,n 

raw_input(’’ \nPress return to exit’’) 

The results of running the program are: 

Integral = -0.894831469504 
nPanels = 64 

It is ciear that Romberg integration is considerably more efficient than the trape- 
zoidal rule—it required 64 panels as compared to 4096 panels for the trapezoidal rule 
in Example 6.4. 

PROBLEM SET6.1 

1. Use the recursive trapezoidal rule to evaluate / 0 3r/4 ln(l + tan x)dx. Explain the 
results. 

2. The table shows the power P supplied to the driving wheels of a car as a function 
of the speed v. If the mass of the car is m= 2000 kg, determine the time At it 
takes for the car to accelerate from 1 m/s to 6 m/s. Use the trapezoidal rule for 
integration. Hint: 

/»6s 

A t=m ( v/P)dv 
Jis 

vvhich can be derived from NevvtorVs law /•’ = m{dv/df) and the definitionof power 
P=Fv. 


v (m/s) 

0 

1.0 

1.8 

2.4 

3.5 

4.4 

5.1 

6.0 

P (kW) 

0 

4.7 

12.2 

19.0 

31.8 

40.1 

43.8 

43.2 


3. Evaluate cos(2cos 1 x)dx with Simpson's 1/3 rule using 2, 4 and 6 panels. 
Explain the results. 

4. Determine /“(1 + x A )~ l dx with the trapezoidal rule using five panels and com¬ 
pare the resuit with the “exact” integral 0.243 75. Hint: use the transformation 
x 3 4 = l/t. 
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6.3 Romberg Integration 



The table below gives the pull F of the bow as a function of the draw x. If the bow 
is drawn 0.5 m, determine the speed of the 0.075-kg arrow when it leaves the bow. 
Hint: the kinetic energy of arrow equals the work done in drawing the bow; that 
is, mv 2 /2 = / 0 °' 5m F dx. 


x (m) 

0.00 

0.05 

0.10 

0.15 

0.20 

0.25 

F (N) 

0 

37 

71 

104 

134 

161 

x (m) 

0.30 

0.35 

0.40 

0.45 

0.50 


F (N) 

185 

207 

225 

239 

250 



6. Evaluate / Q 2 (x 5 + 3x 3 — 2) dx by Romberg integration. 

7. Estimate /J 1 /(x) dx as accurately as possible, where /(x) is defined by the data 


x 

0 

7r/4 

;r/2 

3?r/4 

7r 

/M 

1.0000 

0.3431 

0.2500 

0.3431 

1.0000 


8. Evaluate 


L 


i 


sinx , 

—=r- dx 

y/X 


with Romberg integration. Hint: use transformation of variable to eliminate the 
indeterminacy at x = 0. 

9. Newton-Cotes formulas for evaluating /(x) dx were based on polynomial ap¬ 
proximabo ns of /(x). Show that if y = /(x) is approximated by a natural cubic 
spline with evenly spaced knots at Xo, X\ ,..., x„, the quadrature formula becomes 

h 

I — 2 (To + 2 yi + 2y 2 -|-+ 2y„_i + y„) 

— —— {k 0 + 2fci + k 2 + • • • + 2fc„_i + k n ) 
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where h is the distance between the knots and fc; = y". Note that the first part is 
the composite trapezoidal rule; the second part may be viewed as a “correction” 
for curvature. 

10. ■ Evaluate 

dx 

J o \/ sin v 


with Romberg integration. Hin/.: use the transformation sin x = t 2 . 

11. ■ The periodofa simple pendulum oflengthi is r = AJL/gh{d 0 ), where gis the 
gravitational acceleration, 6o represents the angular amplitude and 


W o) = 


2 

Jl 


de 


y 1 - sin 2 (0 o /2) sin 2 6 


Compute h{ 15°), /i(30 ) and /;(45 ), and compare these values with h[ 0) =n/2 (the 
approximation used for small amplitudes). 



The figure shows an elastic half-space that carries uniform loading of intensity q 
over a circular area of radius a. The vertical displacement of the surface at point 
P can be shown to be 


w{r) = w 0 


r jt/2 

J 0 


cos 2 0 


( r/a) 2 — sin 2 9 


-.de 


r > a 


where is the displacement at r = a. Use numerical integration to determine 
w/wodXr = 2 a. 



13 . 
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6.4 Gaussian Integration 


The mass m is attached to a spring of free length b and stiffness k. The coefficient 
of friction between the mass and die horizontal rod is /x. The acceleration of the 
mass can be shown to be (you may wish to prove this) x = — f(x ), where 

If the mass is released from rest at x = b, its speed at x = 0 is given by 


v 0 = J2 L f[x)dx 

Compute fobynumericalintegrationusingthe data m= 0.8 kg, b = 0.4 m, /x = 0.3, 
k = 80 N/m and g = 9.81 m/s 2 . 

14. ■ Debye’s formula for the heat capacity Cy of a solid is Cy = 9Nkg(u), where 

’i /u 


f L 

g(u) = l? / 

J 0 

The terms in this equation are 


(e* - l ) 2 


dx 


N = number of particles in the solid 
k = Boltzmann constant 
U= T/ ©D 

T = absolute temperature 
©d = Debye temperature 

Compute g(u) from u = 0 to 1.0 in intervals of 0.05 and plot the results. 
15. ■ A power spike in an electric Circuit results in the current 

i{t) = he~ t/h sin(2t/fo) 

across a resistor. The energy E dissipated by the resistor is 


E = f R [i{f)Y dt 
J o 


Find E using the data io = 100 A, R = 0.5 fi and fo = 0.01 s. 


6.4 Gaussian Integration 

Gaussian Integration Formulas 

We found that Newton-Cotes formulas for approximating f{x) dx work best if 
f[x) is a smooth function, such as a polynomial. This is also true for Gaussian 
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quadrature. However, Gaussian formulas are also good at estimating integrals of the 
form 


/ 


w[x) f{x)dx 


(6.15) 


where w[x), called the weighting function, can contain singularities, as long as they 
are integrable. An example of such an integral is /J (1 + x 2 ) ln x dx. Sometimes infinite 
limits, as in / 0 °° e~ x sin x dx, can also be accommodated. 

Gaussian integration formulas have the same form as Newton-Cotes rules: 

n 

I = J2 A if( x i) (6.16) 

1=0 


where, as before, I represents the approximation to the integral in Eq. (6.15). The 
difference lies in the way that the weights A,- and nodal abscissas x,- are determined. In 
Newton-Cotes integration the nodes were evenly spaced in ( a , b), i.e., their locations 
were predetermined. In Gaussian quadrature the nodes and weights are chosen so 
that Eq. (6.16) yields the exact integral if /(x) is a polynomial of degree 2 n+ 1 or less; 
that is, 



n 

w{x)P m {x)dx = E Ai P m {Xi ), 
1=0 


m<2n+ 1 


(6.17) 


One way of determining the weights and abscissas is to substitute P 0 (x) = 1, 
Pi(x) = x,..., P 2 ,i+i(x) = x 2n+1 in Eq. (6.17) and solve the resulting 2n+ 2 equations 


/■b n 

1 w{x)x l dx = Y\ /1 
1=0 


x , 


7 = 0,1,..., 2 n+ 1 


for the unknowns A,- and x,-. 

As an illustration, let w(x) = e~ x , a = 0, b= oo and n= 1. The four equations 
determining Xq, X\, A (i and A, are 



e X dx = A 0 + Ai 
e~ x xdx = AqXq + AiXi 


e X x 2 dx = AqXq + A\x\ 


e X x‘dx = AqXq + Aix 2 
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After evaluating the integrals, we get 

Ao + Ai = 1 
AqXq + AiXi = 1 
AqXq + Aixf = 2 
AqXq + = 6 

The solution is 

Xq = 2 — \f~2 

X\ = 2 + sl~2 

so that the integration formula becomes 

e~ x f{x)dx « [(V2 + 1) /(2 - V 2 ) + (V2 - 1) /(2 + >/ 2 )] 

Due to the nonlinearity of the equations, this approach will not work well for large 
n. Practical methods of finding x, and A,- require some knowledge of orthogonal poly- 
nomials and their relationship to Gaussian quadrature. There are, however, several 
“classical” Gaussian integration formulas for which the abscissas and weights have 
been computed with great precision and tabulated. These formulas can be used with- 
out knowing the theory behind them, since ali one needs for Gaussian integration are 
the values of jc,- and A,-. If you do not intend to venture outside the classical formulas, 
you can skip the next two topics of this chapter. 


Aq — 


Ai = 


V2+ 1 
2V2 
V2- 1 
2V2 


*Orthogonal Polynomials 


Orthogonal polynomials are employed in many areas of mathematics and numerical 
analysis. They have been studied thoroughly and many of their properties are known. 
What follows is a very small compendium of a large topic. 

The polynomials ip n {x), n= 0,1, 2,... (n is the degree of the polynomial) are said 
to form an orthogonal set in the interval (a, h) with respect to the weighting func- 
tion w(x) if 



w{x)(p m [x)(p n {x)dx = 0, 


m/ n 


(6.18) 


The set is determined, except for a constant factor, by the choice of the weighting 
function and the limits of integration. That is, each set of orthogonal polynomials 
is associated with certain w[x), a and b. The constant factor is specified by stan- 
dardization. Some of the classical orthogonal polynomials, named after well-known 
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mathematicians, are listed in Table 6.1. The last column in the table shows the 
standardization used. 


Name 

Symbol 

a 

b 

w(x) 

w(x) [y n {x)\ 2 dx 

Legendre 

Chebyshev 

Laguerre 

Hermite 

Pnix) 

T n (x) 

Lnix) 

H n [x) 

-i 

-i 

0 

—oo 

1 

1 

OO 

oo 

1 

(1 - x 2 )- 1 / 2 

e -x 

-X 2 

e 

2/(2 n+ 1 ) 

? r /2 [n> 0 ) 

1 

s /^i2 n n\ 


Table 6.1 


Orthogonal polynomials obey recurrence relations of the form 

a n (p n+1 (x) = [b n + c n x)(p n {x) - dn<p n _ x (x) (6.19) 

If the lirst two polynomials of the set are known, the other members of the set can be 
computed from Eq. (6.19). The coefficients in the recurrence formula, together with 
cp 0 (x) and ip\ (x), are given in Table 6.2. 


Name 

PoW 

<Pi(x) 

(Z n 

bn 

Cn 

d n 

Legendre 

1 

X 

n+ 1 

0 

2 n+ 1 

n 

Chebyshev 

1 

X 

1 

0 

2 

i 

Laguerre 

1 

1 - X 

n+ 1 

2 n+ 1 

-1 

n 

Hermite 

1 

2x 

1 

0 

2 

2 


Table 6.2 


The classical orthogonal polynomials are also obtainable from the formulas 


(-1)" d n r 2 ,n 
T n (x) = cos(rccos _1 x), n > 0 

pX dn 


d n 


H n (x) = (-1 — 

dx 11 


and their derivatives can be calculated from 

(1 - x 2 )//„(x) = ri{-xp n {x) + Pn- l(x)] 
(1 - x 2 )T' n {x) = n[-xT n {x) + nT„_i(x)] 
xL' n {x) = n[L n (x ) - L„_i(x)] 
H' n (x) = 2nH n -Ax) 


( 6 . 20 ) 


( 6 . 21 ) 
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Other properties of orthogonal polynomials that have relevance to Gaussian in¬ 
tegration are: 


• <p n {x) has n real, distinet zeroes in the interval ( a , b). 

• The zeros of ip n [x) lie between the zeros of < p n+1 [x). 

• Any polynomial P„[x ) of degree ncan be expressed in the form 

n 

Pnix) = J2 CiViW (6.22) 

i =0 


It follows from Eq. (6.22) and the orthogonality property in Eq. (6.18) that 

»b 

w{x)P n {x)(p n+m {x)dx = 0, m> 0 (6.23) 


/ 


*Determination of Nodal Abscissas and Weights 

Theorem The nodal abscissas Xo, Xi,..., x„ are the zeros of the polynomial <p n+1 (x) 
that belongs to the orthogonal set dehned in Eq. (6.18). 


Proof We start the proof by letting /(x) = / J 2 „ + | (x) be a polynomial of degree 2n+ 1. 
Since the Gaussian integration with n+ 1 nodes is exact for this polynomial, we 
have 



n 

W{x)P 2n +l{x)dx = AiP 2 n+\{Xi) 
1=0 


(a) 


A polynomial of degree 2n + 1 can always written in the form 


P 2 ,h-iW = Q„(x) + RnWy^ix) 


(b) 


where Q n (x), R n [x) and <p n+t (x) are polynomials of the degree indicated by the 
subscripts. 14 Therefore, 

/»b pb pb 

/ w[x)P 2tVr i{x)dx = / w{x)Q n [x)dx + / w{x)R n [x)(p n+l {x)dx 
J a J a J a 

But according to Eq. (6.23) the second integral on the right-hand side vanishes, 

so that 


pb p b 

/ w{x)P 2n+ i[x)dx = / w{x)Q n {x)dx 
J a J a 


(C) 


Because a polynomial of degree /i is uniquely defined by n + 1 points, it is always 
possible to find Aj such that 

-b _ 

(d) 
;=o 


pb n 

/ iU(x) Q n {x)dx =Y A i Qn(Xi) 

J a ;_n 


14 


It can be shown that Q n [x) and R n (x ) are unique for given P 2n ^\ (x) and <p n+1 (x). 
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In order to arrive at Eq. (a), we must choose for the nodal abscissas x,- the roots 
of <p n+l {x) = 0. According to Eq. (b) we then have 

P 2 n+l(Xi) = Q„(X,), i = 0, 1. H (e) 

which together with Eqs. (c) and (d) leads to 

r*h tt 

/ W[x)P 2 n+l{x)dx = / w{x)Q n {x)dx = E A-i P2n+ 1 {%i) 

J Cl J & f—Q 

This completes the proof. 


Theorem 


Ai = ( w{x)li{x)dx t i = 0,l,...,n (6.24) 

J a 


where £,( x) are the Lagrange’s Cardinal functions spanning the nodes at 
Xq,Xi, ... x n . These functions were defined in Eq. (3.2). 


Proof Applying Lagrange’s formula, Eq. (3.1a), to Q n [x) yields 

n 

Qnix) = QniXiUiW 

i =0 

which upon substitution in Eq. (d) gives us 


n r /» 

E! QniXi) I 

i =0 L Ja 


QniXi) / w{x)li[x)dx 


= Y AiQniXi) 


i =0 


or 


ti v r 

Y Qn(Xi) Ai - / 
i'=0 L Ja 


u>(_x)lilx)dx 


= 0 


This equation can be satisfied for arbitrary Q(x) of degree nonly if 

f b 

A{— / w{x)li{x)dx = 0, i = 0, \,...,n 
J a 

which is equivalent to Eq. (6.24). 

It is not difficult to compute the zeros x,-, i = 0,1..... n of a polynomial <p n+ , (x) 
belonging to an orthogonal set by one of the methods discussed in Chapter 4. Once 
the zeros are known, the weights A,-, i = 0,1,..., n could be found from Eq. (6.24). 
However the following formulas (given without proof) are easier to compute 
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Gauss-Legendre 

Gauss-Laguerre 

Gauss-Hermite 


2 


X i [L' n+ \ ( x 0] 
2 n+2 {n+ 

KiWf 


(6.25) 


Abscissas and Weights for Classical Gaussian Quadratures 

Here we list some classical Gaussian integration formulas. The tables of nodal ab¬ 
scissas and weights, covering n = 1 to 5, have been rounded off to six decimal places. 
These tables should be adequate for hand computation, but in programming you 
may need more precision or a larger number of nodes. In that case you should consuit 
other references, 15 or use a subroutine to compute the abscissas and weights within 
the integration program. 16 

The truncation error in Gaussian quadrature 

nb n 

E = w(x) f[x)dx - Y A i /(*;) 

i =0 

has the form E = K{ri) f 2n+2) {c), where a < c < b (the value of c is unknown; only 
its bounds are given). The expression for K[n) depends on the particular quadrature 
being used. If the derivatives of f{x) can be evaluated, the error formulas are useful 
in estimating the error bounds. 

Gauss-Legendre Quadrature 

C md%~Y A i mi) ( 6 . 26 ) 

■’~ l ;=o 


-H 

±li At 

n= 1 

0.577 350 1.000 000 

n= 2 

0.000 000 0.888 889 

0.774 597 0.555 556 

n= 3 

0.339 981 0.652145 

0.861136 0.347 855 

n= 4 

0.000 000 0.568 889 

0.538469 0.478 629 

0.906180 0.236927 

n= 5 

0.238 619 0.467 914 

0.661209 0.360 762 

0.932 470 0.171324 


Table 6.3 


15 Abramowitz, M., and Stegun, I. A., Handbook ofMathematicalFunctions, Dover Publications, 1965; 
Stroud, A. H., and Secrest, D., Gaussian Quadrature Formulas, Prentice-Hall, 1966. 

16 Several such subroutines are listed in W. H. Press et al., Numerical Recipes in Fortran 90, Cambridge 
University Press, 1996. 
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This is the most often used Gaussian integration formula. The nodes are arranged 
symmetrically about § = 0, and the weights associated with a symmetric pair of nodes 
are equal. For example, for n = 1 we have | 0 = —and A (} = A t . The truncation error 
in Eq. (6.26) is 


E = 




(2n+ 3) [(2/1+2)!] 


3 J 


(6.27) 


To apply Gauss-Legendre quadrature to the integral f{x) dx, we must first map 
the integration range ( a , b ) into the “Standard” range (—1,1). We can accomplish this 
by the transformation 


x = 


b+ a 


+ 


b- a 
2 


§ 


(6.28) 


Now dx= d%{b— a)/ 2, and the quadrature becomes 



b— a 
2 


Z >/(*0 

i=0 


(6.29) 


where the abscissas jc,- must be computed from Eq. (6.28). The truncation error 
here is 


E = (b-d)^l{n+m 4 f2n+2)[c] 

(2n+ 3) l(2n+ 2)!] 3 J 


a < c < b 


(6.30) 


Gauss-Chebyshev Quadrature 


/ (! — x 2 ) 1/2 f{x)dx ~ fixi) 

J -1 " +1 ^ 


(6.31) 


Note that all the weights are equal: Ai = tt/ (n+ 1). The abscissas of the nodes, which 
are symmetric about x = 0, are given by 


Xi 


= cos 


(2 i + l);r 
2n+2 


(6.32) 


The truncation error is 


E = 


2ji 

2 2n+2 (2n+ 2)! 


f 2n+2 \c), 


- 1 < c < 1 


(6.33) 


Gauss-Laguerre Quadrature 


/»oo n 

/ e~ x f{x) dx^Y A ‘ /fe 

Jo i'=0 


(6.34) 
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Xi 


At 

Xi 


A; 


n = 1 



n= 4 


0.585 786 


0.853 554 

0.263 560 


0.521756 

3.414214 


0.146447 

1.413403 


0.398 667 


n= 2 


3.596426 


(-1)0.759 424 

0.415 775 


0.711093 

7.085 810 


(-2)0.361175 

2.294 280 


0.278 517 

12.640 801 


(-4)0.233 670 

6.289 945 


(-1)0.103 892 


n= 5 



n= 3 


0.222 847 


0.458964 

0.322 548 


0.603 154 

1.188 932 


0.417 000 

1.745 761 


0.357418 

2.992 736 


0.113373 

4.536 620 


(-1)0.388 791 

5.775 144 


(-1)0.103 992 

9.395 071 


(-3)0.539 295 

9.837 467 


(-3)0.261017 




15.982 874 


(-6)0.898 548 


Table 6.4. Multiply numbers by 10*-', where /c is given in parentheses 


° <c< ” (6 - 35 > 


Gauss-Hermite Quadrature 



n 

e~ x2 f{x)dx « Y A i /te) 
!=0 


(6.36) 


The nodes are placed symmetrically about x = 0, each symmetric pair having the 
same weight. 


-H 

±Xi Ai 

n= 1 

0.707107 0.886227 

n= 2 

0.000 000 1.181636 

1.224745 0.295 409 

n= 3 

0.524 648 0.804 914 

1.650 680 (-1)0.813128 

n = 4 

0.000 000 0.945 308 

0.958 572 0.393 619 

2.020183 (-1)0.199 532 

n= 5 

0.436077 0.724 629 

1.335 849 0.157 067 

2.350 605 (-2)0.453 001 


Table 6.5. Multiply numbers by 10 fc , where k is given in parentheses 


E = 


\fx{ n + 1)! K 2n+2) (r) 

2 2 {2n + 2)! J 


0 < c < oo 


(6.37) 
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Gauss Quadrature with Logarithmic Singularity 


1 n 

fix) ln [x)dx ~ - ^ Aif[Xi ) (6.38) 

i=0 


Xi Ai 

Xi Ai 

n= 1 

0.112 009 0.718 539 

0.602 277 0.281461 

n= 2 

(-1)0.638 907 0.513 405 

0.368 997 0.391980 

0.766880 (-1)0.946154 

n= 3 

(-1)0.414 485 0.383 464 

0.245 275 0.386 875 

0.556165 0.190435 

0.848 982 (-1)0.392255 

n= 4 

(-1)0.291345 0.297 893 

0.173 977 0.349 776 

0.411703 0.234488 

0.677314 (-1)0.989305 

0.894 771 (-1)0.189 116 

n= 5 

(-1)0.216344 0.238764 

0.129583 0.308287 

0.314 020 0.245 317 

0.538 657 0.142 009 

0.756916 (-1)0.554 546 

0.922 669 (-1)0.101690 


Table 6.6. Multiply numbers by 1 0 k , where Ic is given in parentheses 


E = - -- f 2n+1) ic), 0 < c < 1 (6.39) 

(2n+l)r 

where k{ 1) = 0.00 285, k{ 2) = 0.000 17, k{ 3) = 0.00001. 

■ gaussNodes 

The function gaussNodes listed below 17 computes the nodal abscissas x, and the 
corresponding weights Aj used in Gauss-Legendre quadrature over the “Standard" 
interval (—1,1). It can be shown that the approximate values of the abscissas are 

n[i + 0.75) 

Xi = cos - 

m+ 0.5 

where m= n+ 1 is the number of nodes, also called the integration order. Using these 
approximations as the starting values, the nodal abscissas are computedby findingthe 
nonnegative zeros of the Legendre polynomial p m (x) with Newton’s method (the neg¬ 
ative zeros are obtained from symmetry). Note that gaus sNode s calls the subfunction 
legendre, which returns p m {t) and its derivative as the tuple (p, dp). 

17 This function is an adaptation of a routine in Press, W. H. et al., Numerical Recipes in Fortran 90, 
Cambridge University Press, 1996. 
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## module gaussNodes 

’’’ x,A = gaussNodes(m,tol=10e-9) 

Returns nodal abscissas {x> and weights {A} of 
Gauss-Legendre m-point quadrature. 

from math import cos,pi 

from numarray import zeros,Float64 

def gaussNodes(m,tol=10e-9): 

def legendre(t,m): 

pO = 1.0; pl = t 
for k in range(l,m): 

p = ((2.0*k + 1.0)*t*pl - k*p0)/(l.0 + k) 
pO = pl; pl = p 

dp = m*(pO - t*pl)/(1.0 - t**2) 
return p,dp 

A = zeros((m),type=Float64) 
x = zeros((m),type=Float64) 

nRoots = (m + l)/2 # Number of non-neg. roots 

for i in range(nRoots): 

t = cos(pi*(i + 0.75)/(m + 0.5)) # Approx. root 

for j in range(30): 

p,dp = legendre(t,m) # Newton-Raphson 

dt = -p/dp; t = t + dt # method 

if abs(dt) < tol: 

x[i] = t; x[m-i-l] = -t 

A[i] = 2.0/(1.0 - t * * 2)/(dp * * 2) # Eq.(6.25) 

A[m-i-l] = A[i] 

break 

return x,A 

■ gaussQuad 

The function gaussQuad utilizes gaussNodes to evaluate f f(x) dx with Gauss- 
Legendre quadrature using m nodes. The function routine for /( x) must be supplied 
by the user. 

## module gaussQuad 

’’’ I = gaussQuad(f,a,b,m). 

Computes the integral of f(x) from x = a to b 
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with Gauss-Legendre quadrature using m nodes. 

from gaussNodes import * 

def gaussQuad(f,a,b,m): 
cl = (b + a)/2.0 
c2 = (b - a)/2.0 
x,A = gaussNodes(m) 
sum = 0.0 

for i in range(len(x)): 


sum = sum + A[i]*f(cl + c2*x[i]) 
return c2*sum 


EXAMPLE 6.8 


Evaluate (1 — x 2 ) 3/2 dx as accurately as possible with Gaussian integration. 


Solution As the integrand is smooth and free of singularities, we could use Gauss- 
Legendre quadrature. However, the exact integral can obtained with the Gauss- 
Chebyshev formula. We write 



The numerator /(x) = (1 — x 2 ) 2 is a polynomial of degree four, so that Gauss- 
Chebyshev quadrature is exact with three nodes. 

The abscissas of the nodes are obtained from Eq. (6.32). Substituting n= 2, 
we get 


Xi = cos 


6 


Therefore, 


7r \/3 

Xo = cos - = — 
6 2 


Xi = cos — = 0 
2 


5 7T V3 

x? = cos — = — 
6 2 


and Eq. (6.31) yields 
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EXAMPLE 6.9 

Use Gaussian integration to evaluate / 0 ° J cos itx ln x dx. 


Solution We split the integral into two parts: 

» 0.5 


/ 


f 


cos7rxlnx dx = / cos7rxlnxdx — / cos7rxlnxdx 


/■ 


The first integral on the right-hand side, which contains a logarithmic singularity at 
x = 0, can be computed with the special Gaussian quadrature in Eq. (6.38). Choosing 
n = 3, we have 


/' 


cosjrxlnxdx: 


3 

— ^ Ai cos jrx,- 
i=0 


The sum is evaluated in the following table: 


Xi 

cosjtXi 

Ai 

Ai cos ;r Xi 

0.041448 

0.991534 

0.383 464 

0.380 218 

0.245 275 

0.717 525 

0.386 875 

0.277 592 

0.556 165 

-0.175533 

0.190435 

-0.033 428 

0.848 982 

-0.889 550 

0.039 225 

-0.034 892 

E = 0.589 490 


Thus 


f 


cosjrxlnxdx ~ —0.589490 


The second integral is free of singularities, so that it can be evaluated with Gauss- 
Legendre quadrature. Choosing n= 3, we have 

3 


/»1 3 

I cos^-xlnxdx « 0.25 Aj cos jrx^lnx,- 

J 0.5 i =0 


where the nodal abscissas are (see Eq. (6.28)) 

1 + 0.5 1 - 0.5 


x,- = 


+ 


?i = 0.75+ 0.25? * 


2 2 

Looking up and Ai in Table 6.3 leads to the following computations: 


Ii 

Xi 

COS7TX;lnX,- 

Ai 

At cos jrXi ln Xi 

-0.861136 

0.534716 

0.068141 

0.347 855 

0.023 703 

-0.339 981 

0.665 005 

0.202 133 

0.652 145 

0.131820 

0.339 981 

0.834 995 

0.156638 

0.652 145 

0.102151 

0.861136 

0.965 284 

0.035 123 

0.347 855 

0.012218 

E = 0.269 892 
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from which 


Therefore, 


/'■ 


cosnxlnxdxtt 0.25(0.269 892) = 0.067 473 


f 


cos 7tx ln x dx « —0. 589 490 — 0.067 473 = —0. 656 96 3 
which is correct to six decimal places. 


EXAMPLE 6.10 

Evaluate as accurately as possible 


F = 



x T- 3 
■s/x 


e X dx 


Solutiori In its present form, the integral is not suited to any ofthe Gaussian quadra- 
tures listed in this chapter. But using the transformation 


x = i 2 dx = 2 tdt 


we have 

/* OO /*oo 

F = 2 (t 2 + 3)e~^dt = / [f + 3)e~'~dt 

J 0 J —oo 

which can be evaluated exactly with Gauss-Hermite formula using only two nodes 
( n= 1). Thus 

F= Aoi^ + 3) + A 1 tf + 3) 

= 0.886227 [(0.707 107) 2 + 3] + 0.886227 [(-0.707 107) 2 + 3] 

= 6.203 59 


EXAMPLE 6.11 

Determine how many nodes are required to evaluate 



with Gauss-Legendre quadrature to six decimal places. The exact integral, rounded 
to six places, is 1.418 15. 


Solution The integrand is a smooth function; hence it is suited for Gauss-Legendre 
integration. There is an indeterminacy at x = 0, but this does not bother the quadra¬ 
ture since the integrand is never evaluated at that point. We used the following pro- 
gram that computes the quadrature with 2,3,... nodes until the desired accuracy is 
reached: 
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## example 6_11 

from math import pi,sin 

from gaussQuad import * 

def f(x): return (sin(x)/x)**2 

a = 0.0; b = pi; 

Iexact = 1.41815 
for m in range(2,12): 

I = gaussQuad(f,a,b,m) 
if abs(I - Iexact) < 0.00001: 

print ’'Number of nodes = ’ ’ ,m 

print ’’Integral gaussQuad(f,a,b,m) 

break 

raw_input(’’ \nPress return to exit’’) 

The program output is 

Number of nodes = 5 
Integral = 1.41815026778 

EXAMPLE 6.12 

Evaluate numerically f* 5 f(x) dx, where f{x) is represented by the unevenly spaced 
data 


X 

1.2 

1.7 

2.0 

2.4 

2.9 

3.3 

/M 

-0.36236 

0.128 84 

0.41615 

0.737 39 

0.97096 

0.987 48 


Knowing that the data points lie on the curve /(x) = — cos x, evaluate the accuracy of 
the solution. 


Solutiori We approximate /(x) by the polynomial R,(x) that intersects all the data 
points, and then evaluate f^ 5 /(x) dx ~ J -3 P 5 (x)dxwith the Gauss-Legendre formula. 
Since the polynomial is of degree five, only three nodes {n = 2) are required in the 
quadrature. 

From Eq. (6.28) and Table 6.3, we obtain for the abscissas of the nodes 


x 0 = 

Xi = 

*2 = 


3+1.5 

2 

3+1.5 

2 

3+1.5 

2 


3- 1.5 

+ —-—(-0.774597) = 1.6691 

= 2.25 

3- 1.5 

+ —-— 


(0.774597) = 2.8309 
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We now compute the values of the interpolant P 5 U) at the nodes. This can be done 
using the modules newtonPoly or neville listed in Section 3.2. The results are 

P 5 U 0 ) = 0.098 08 P 5 (x 1 ) = 0.62816 P 5 (x 2 ) = 0.952 16 

From Gauss-Legendre quadrature 

I = [ P 5 (x) dx = 3 15 V Ai P 5 (x,-) 

i =0 

we get 

/ = 0.75 [0.555 556(0.098 08) + 0.888 889(0.628 16) + 0.555 556(0.952 16)] 

= 0.85637 

Comparison with — f* 5 cos xdx= 0.856 38 shows that the discrepancy is within the 
roundoff error. 


PROBLEM SET6.2 

1. Evaluate 

lnx 

- dx 

- 2x + 2 

with Gauss-Legendre quadrature. Use (a) two nodes and (b) four nodes. 

2. Use Gauss-Laguerre quadrature to evaluate / 0 °°(1 - x 2 ) 3 e~ x dx. 

3. Use Gauss-Chebyshev quadrature with six nodes to evaluate 




dx 

v/sinx 


Compare the resuit with the “exact” value 2.62206. Hint : substitute sinx = i 2 . 

4. The integral / 0 * sin x dx is evaluated with Gauss-Legendre quadrature using four 
nodes. What are the bounds on the truncation error resuiting from the quadrature? 

5. How many nodes are required in Gauss-Laguerre quadrature to evaluate / 0 °° e~ x 
sin x dx to six decimal places? 

6 . Evaluate as accurately as possible 

r 1 2x +1 , 

/ - dx 

J 0 V*(l - x) 


Hint: substitute x = (1 + f)/2. 

7. Compute f* sin x ln x dx to four decimal places. 

8 . Calculate the bounds on the truncation error if fg xsinxdx is evaluated with 
Gauss-Legendre quadrature using three nodes. What is the actual error? 
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9. Evaluate / Q 2 (sinh x/x) dx to four decimal places. 

10. Evaluate the integral 

r°° xdx 
Jo e* + l 

to six decimal places. Hint: substitute e* = 1/t. 

11. ■ The equation of an ellipse is jc 2 /^ 2 + y 2 //^ = l.Write a programthat computes 
the length 


S = 



\/l + ( dy/dx) 2 dx 


of the circumference to five decimal places for given a and b. Test the program 
with a = 2 and b= 1. 

12. ■ The error function, which is of importance in statistics, is defined as 

2 r x 

erf(x) = —— / e~’dL 
4* Jo 

Write a program that uses Gauss-Legendre quadrature to evaluate erf(x) for a 
given x to six decimal places. Note that erf(x) = 1.000 000 (correct to 6 decimal 
places) when x > 5. Test the program by verifying that erf(l.O) = 0.842 701. 



The sliding weight of mass m is attached to a spring of stiffness k that has an 
undeformed length L. When the mass is released from rest at B, the time it takes 
to reach A can be shown to be t = C jlnpc, where 




- 1/2 


dz 


Compute C to six decimal places. Hint: the integrand has a singularity at z = 1 
that behaves as (1 — 
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14. ■ 



x 


A 


P 


h 


y 


A uniform beam forms the semiparabolic cantilever arch AB. The vertical dis- 
placement of A due to the force P can be shown to be 



where EI is the bending rigidity of the beam and 



Write a program that computes C(h/b) for any given value of h/b to four decimal 
places. Use the program to compute C(0.5), C(1.0) and C(2.0). 


■ There is no elegant waytocompute/ = fy 2 1 n (si n x) dx. A “brute force” method 


15. 


that works is to split the integral into several parts: from x = 0 to 0.01, from 0.01 
to 0.2 and from x = 0.2 to jr/2. In the first part we can use the approximation 
sin x « x, which allows us to obtain the integral analytically. The other two parts 
can be evaluated with Gauss-Legendre quadrature. Use this method to evaluate 
I to six decimal places. 


16. ■ 



The pressure of wind was measured at various heights on a vertical wall, as shown 
on the diagram. Find the height of the pressure center, which is defined as 


fj l2m h p(h) dii 

k= r 77 ;- 

/ 0 m p(M dh 
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Hint: fit a cubic polynomial to the data and then apply Gauss-Legendre 
quadrature. 


*6.5 Multiple Integrals 


Multiple integrals, such as the area integral f f A f(x, y) dx dy, can also be evaluated 
by quadrature. The computations are straightforward if the region of integration has a 
simple geometric shape, such as a triangle or a quadrilateral. Due to complications in 
specifying the limits of integration on x and y, quadrature is not a practical means of 
evaluating integrals over irregular regions. However, an irregular region A can always 
be approximated as an assembly of triangular or quadrilateral subregions A t , A 2 ,, 
called finite elements, as illustrated in Fig. 6.6. The integral over Acan thenbe evaluated 
by summing the integrals over the finite elements: 


/ fjfoyidxdy* ?//* 


f[x, y) dx dy 


Volume integrals can computed in a similar manner, using tetrahedra or rectangular 
prisms for the finite elements. 



Boundary of region A 


Figure 6.6. Finite element model of an irregular 
region. 


Gauss-Legendre Quadrature over a Quadrilateral Element 



5 = 1 


Figure 6.7. Mapping a quadrilateral into the Standard rectangle. 
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Consider the double integral 

1 = J J ^ /(?, dn 

over the rectangular element shown in Fig. 6.7(a). Evaluating each integral in turn by 
Gauss-Legendre quadrature using n+ 1 integration points in each coordinate direc- 
tion, we obtain 

/ 1 n n 

^ d, l = A i 
■1 ;=n i=n 


I=J^J^ A i A jf^i,r,j) (6.40) 

i '=0 7=0 

As noted previously, the number of integration points in each coordinate direction, 
m= n+ 1, is called the integration order. Figure 6.7(a) shows the locations of the 
integration points used in third-order integration {m= 3). Because the integration 
limits were the “Standard” limits (—1,1) of Gauss-Legendre quadrature, the weights 
and the coordinates of the integration points are as listed Table 6.3. 

In order to apply quadrature to the quadrilateral element in Fig. 6.7(b), we must 
first map the quadrilateral into the “Standard” rectangle in Fig. 6.7(a). By mapping 
we mean a coordinate transformation x = jc(£, rf), y = y(f, 77 ) that results in one-to- 
one correspondence between points in the quadrilateral and in the rectangle. The 
transformation that does the job is 

4 4 

X^,r t ) = Y / N k^,r 1 )x k y$,ii) = Y J NM,n)yk (6.41) 

k=l k= 1 




i '=0 


where {x k , y k ) are the coordinates of corner k of the quadrilateral and 

A/lM = ^(l-|)(l-i?) 

AkM = i(l+$)(l-i,) (6.42) 

= Ja+f)(i + )7) 

A4M = jd-Dd + ij) 

The functions Afc(|, j/), known as the shape functions, are bilinear (linear in each 
coordinate). Consequently, straight lines remain straight upon mapping. In particular, 
note that the sides of the quadrilateral are mapped into the lines f = ±1 and rj = ± 1 . 
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Because mapping distorts areas, an infinitesimal area element d/\ = dx dy of the 
quadrilateral is not equal to its counterpart dA' = dq of the rectangle. It can be 
shown that the relationship between the areas is 


dxdy= |J(§, 77 ) | d% dq 


(6.43) 


where 


JG. rj) = 


- dX 

9JH 

9 ? 


dX 

9y 

- d>l 

dq _ 


(6.44a) 


is known as the Jacobian matrix of the mapping. Substituting from Eqs. (6.41) and 
(6.42) and differentiating, we find that the components of the Jacobian matrix are 

1 


/ll — 7 [— ( 1 — rj)x 1 + (1 — q)X 2 + (1 + q)x% — (1 — q)x^] 


J 12 = - [-(1 - T])yi + (1 - rj)yz + (1 + >7)3/3 - (1 - >7)34] 


/21 — “ [— (1 — ?)-tl — (1 + f)X2 + (1 + £)-*3 + (1 — ^)X4] 


(6.44b) 


/22 = - [-(i - ?)yi - (1 + + (1 + £)j*s + (1 - 1 ) 34 ] 


We can now write 


J J f(x,y)dxdy=J J /[x(| , 77 ), q)] |J(f, q)\d% dq (6.45) 

Since the right-hand-side integral is taken over the “Standard” rectangle, it can be 
evaluated using Eq. (6.40). Replacing /(£, q) inEq. (6.40) bytheintegrandinEq. (6.45), 
we get the following formula for Gauss-Legendre quadrature over a quadrilateral 
region: 


/ = EE A ‘ A J /[*««• Mt’ rij )| 

1=0 7=0 


(6.46) 


The § and ;?-coordinates of the integration points and the weights can again be ob- 
tained from Table 6.3. 


■ gaussQuad2 

The function gaussQuad2 in this module computes f f A f(x. y) dxdy over a quadri¬ 
lateral element with Gauss-Legendre quadrature of integration order m. The quadri¬ 
lateral is dehned by the arrays x and y, which contain the coordinates of the four 
corners ordered in a counterclockwise directiori around the element. The determinant 
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of the Jacobian matrix is obtained by calling the function j ac; mapping is performed 
by map. The weights and the values of | and rj at the integration points are computed 
by gaussNodes listed in the previous article (note that § and rj appear as s and t in the 
listing). 

## module gaussQuad2 

’’’ I = gaussQuad2(f,xc,yc,m). 

Gauss-Legendre integration of f(x,y) over a 
quadrilateral using integration order m. 

{xc},{yc} are the corner coordinates of the quadrilateral. 

from gaussNodes import * 

from numarray import zeros,Float64,dot 


def gaussQuad2(f,x,y,m): 

def jac(x,y,s,t): 

J = zeros((2,2),type=Float64) 

J[0,0] = -(1.0 - t)*x[0] + (1.0 - t)*x[l] \ 

+ (1.0 + t)*x[2] - (1.0 + t)*x[3] 

J[0,1] = -d.O - t)*y[0] + (1.0 - t)*y[ 1] \ 

+ (1.0 + t)*y[2] - (1.0 + t)*y[3] 

J[1,0] = -(1.0 - s)*x[0] - (1.0 + s)*x[l] \ 

+ (1.0 + s)*x[2] + (1.0 - s)*x[3] 

J[l,l] = -(1.0 - s)*y[0] - (1.0 + s)*y[1] \ 

+ (1.0 + s)*y[2] + (1.0 - s)*y[3] 

return (J[0,0]*J[1,1] - J[0,1]*J[1,0])/16.0 

def map(x,y,s,t): 

N = zeros((4),type=Float64) 

N[0] = (1.0 - s)*(l.0 - t)/4.0 

N[1] = (1.0 + s)*(l.0 - t)/4.0 

N[2] = (1.0 + s)*(1.0 + t)/4.0 

N[3] = (1.0 - s)*(l.0 + t)/4.0 

xCoord = dot(N,x) 
yCoord = dot(N,y) 
return xCoord,yCoord 


s,A = gaussNodes(m) 


sum 


0.0 
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for i in range(m): 

for j in range(m): 

xCoord,yCoord = map(x,y,s[i],s[j]) 
sum = sum + A[i]*A[j ] * jac(x , y , s[ i], s[j]) 
*f(xCoord,yCoord) 


return sum 


\ 


EXAMPLE 6.13 



Evaluate the integral 


I = 


J J (x 2 + y) dx dy 


analytically by first transforming it from the quadrilateral region A shown to the “Stan¬ 
dard” rectangle. 


Solution The corner coordinates of the quadrilateral are 


x J = [o 2 2 o] y r = [o 0 3 2 ] 


The mapping is 


x($, rj) = ri)x k 


k= 1 

= o + ( 1 + g)a-.) (2]+ a + ^i + .) (2) + 0 
= 1 + 1 

4 

y(t, ri) = Y AfeCf, rflyt 

k= 1 

(1+1X1 + ?)) (l-IKl + J?) 

= 0 + 0+ --—-— (3) + --—-—(2) 


(5 + £)(! + 17 ) 


4 
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which yields for the Jacobian matrix 

JM = 

Thus the area scale factor is 


‘ dX 

9JH 



1 + 3 

9f 

9? 


1 

4 

3x 

sy 


0 

5 + § 

_ drj 

9 77 _ 



4 J 


IJG,i/)l = 


5 + ? 


Now we can map the integral from the quadrilateral to the Standard rectangle. Refer- 
ring to Eq. (6.45), we obtain 


/:/: 


d+?) 2 + 


(5 + $)(! + 17 ) 


5 + ? 


dl; d?l 


1 ^ '45 27 29 2 1 3 25 5 v 1 2 . , 

777 H + 777? + 7? + 777'i + 77?3 + 77 :?V % dri 


-1 -1 16 8 


16 


16 


8 


16 


Noting that only even powers of £ and ?; contribute to the integral, we can simplify the 
integral to 

.1 ,,1 


I = 




16 16 


EXAMPLE 6.14 

Evaluate the integral 


f 1 f 1 nx ny 

/ / cos — cos — dx dy 

7 -i y-i 2 2 7 

by Gauss-Legendre quadrature of order three. 

Solution From the quadrature formula in Eq. (6.40), we have 


2 2 


1 , COS ^ cos 72 




1=0 7=0 



The integration points are shown in the figure; their coordinates and the correspond- 
ing weights are listed in Table 6.3. Note that the integrand, the integration points and 
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the weights are all symmetric about the coordinate axes. It follows that the points 
labeled a contribute equal amounts to /; the same is true for the points labeled b. 
Therefore, 


I = 


4(0.555 556) cos 


2 2 jr (0.774 597) 


it (0.774 597) 

+ 4(0.555 556) (0.888 889) cos —-- cos 


■> ■> 7r(0) 

+ (0.888 889) 2 cos 2 


1.623 391 


7T(0) 

2 


The exact value of the integral is 16/jr 2 ~ 1.621139. 

EXAMPLE 6.15 



Utilize gaussQuad2 to evaluate I = J f f(x, y ) dxdy over the quadrilateral shown, 
where 

f{x,y)= (x-2f(y-2) 2 

Use enough integration points for an “exact” answer. 

Solution The required integration order is determined by the integrand in 
Eq. (6.45): 

1= /1 /!^ ii(f ’ 11)1 ^ drt (a) 
We note that |/ (f, ri)\, defined in Eqs. (6.44), is biquadratic. Since the specified f[x, y ) 
is also biquadratic, the integrand in Eq. (a) is a polynomial of degree 4 in both £ and 
rj. Thus third-order integration is sufficient for an “exact” resuit. 

#!/usr/bin/python 
## example 6_15 
from gaussQuad2 import * 
from numarray import array 
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def f(x,y): return ((x - 2.0)**2)*((y - 2.0)**2) 

x = array([0.0, 4.0, 4.0, 1.0]) 
y = array([0.0, 1.0, 4.0, 3.0]) 

m = eval(raw_input(’’Integration order ==> ’’)) 
print ’’Integral =’’, gaussQuad2(f,x,y,m) 
raw_input(’’ \nPress return to exit’’ 

Running the above program produced the following resuit: 

Integration order ==> 3 
Integral = 11.3777777778 


Quadrature over a Triangular Element 



A triangle may be viewed as a degenerate quadrilateral with two of its corners 
occupying the same location, as illustrated in Fig. 6.8. Therefore, the integration for¬ 
mulas over a quadrilateral region can also be used for a triangular element. However, 
it is computationally advantageous to use integration formulas specially developed 
for triangles, which we present without derivation. 18 





Figure 6.9. Triangular element. 


© 


Consider the triangular element in Fig. 6.9. Drawing straight lines from the point 
P in the triangle to each of the corners, we divide the triangle into three parts with 
areas Ai, A 2 and A 3 . The so-called area coordinates of P are defined as 



(6.47) 


18 


The triangle formulas are extensively used in the finite method analysis. See, for example, 
Zienkiewicz, O. C., and Taylor, R. L., The Finite Element Method, Vol. 1, 4th ed., McGraw-Hill, 
1989. 
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where A is the area of the element. Since A, + A 2 + A :i = A, the area coordinated are 
related by 

«i + «2 + a 3 = 1 (6.48) 


Note thata,- ranges from 0 (when P lies on the side opposite to corner i) to 1 (when P 
is at corner i). 

A convenient formula of computing A from the corner coordinates (jq, y,) is 


4 1 1 1 1 

A = 2 Xl X2 X3 

yi yi y) 

The area coordinates are mapped into the Cartesian coordinates by 

3 3 

xioti,012,013) = '^aiXi y[oii, a 2 , 013) = 

i= 1 i=l 

The integration formula over the element is 

J jf /[*(«), y(a)] dA=Aj 2 W k f[x(a k ), y(a k )] 


(6.49) 


(6.50) 


(6.51) 


where a k represents the area coordinates of the integration point k, and W k are the 
weights. The locations of the integration points are shown in Fig. 6.10, and the corre- 
sponding values of a k and V14 are listed in Table 6.7. The quadrature in Eq. (6.51) is 
exact if /( x, y) is a polynomial of the degree indicated. 



(a) Linear 



(b) Quadratio 



(c) Cubic 


Figure 6.10. Integration points of trian- 
gular elements. 


Degree of /( x, y) 

Point 

a k 

Wt 

(a) Linear 

a 

1/3, 1/3, 1/3 

1 

(b) Quadratic 

a 

1/2,0, 1/2 

1/3 


b 

1/2, 1/2, 0 

1/3 


c 

0, 1/2, 1/2 

1/3 

(c) Cubic 

a 

1/3, 1/3, 1/3 

-27/48 


b 

1/5, 1/5, 3/5 

25/48 


C 

3/5.1/5, 1/5 

25/48 


d 

1/5, 3/5, 1/5 

25/48 


Table 6.7 
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■ triangleQuad 

The function triangleQuad computes / f A /(x, y) dx dy over a triangular region us- 
ing the cubic formula—case (c) in Fig. 6.10. The triangle is defined by its corner coor- 
dinate arrays xc and yc, where the coordinates are listed in a counterclockwise order 
around the triangle. 

## module triangleQuad 

’’’ I = triangleQuad(f,xc,yc). 

Integration of f(x,y) over a triangle using 
the cubic formula. 

{xc},{yc} are the corner coordinates of the triangle. 
from numarray import array,matrixmultiply 
def triangleQuad(f,xc,yc): 


alpha = arrayC[[1.0/3, 1.0/3.0, 1.0/3.0], \ 


[ 0 . 2 , 0 . 2 , 0 . 6 ], 

[ 0 . 6 , 0 . 2 , 0 . 2 ], 

[ 0 . 2 , 0 . 6 , 0 . 2 ]]) 


\ 

\ 


W = array([-27.0/48.0 ,25.0/48.0, 25.0/48.0, 25.0/48.0]) 
x = matrixmultiplyCalpha,xc) 
y = matrixmultiplyCalpha,yc) 


A = (xc[l]*yc[2] - xc[2]*yc[l] 
- xc[0]*yc[2] + xc[2]*yc[0] 


\ 

\ 


+ xc[0]*yc[l] - xc[1]*yc[0])/2.0 
sum = 0.0 
for i in range(4): 

sum = sum + W[i] * f(x[i],y[i]) 
return A*sum 


EXAMPLE 6.16 


© " 



© 
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Evaluate I = JJ J\x, y) dx dy over the equilateral triangle shown, where 19 

/U, y) = I(x 2 + y 2 ) - i(x 3 - 3xf) - \ 

2 b 3 

Use the quadrature formulas for (1) a quadrilateral; and (2) a triangle. 

Solution of Part (1) Let the triangle be formed by collapsing corners 3 and 4 of a 
quadrilateral. The corner^coordinates of this quadrilateral are x= [—1, —1, 2, 2 ] T 
and y = - Vs, 0, oj . To determine the minimum required integration order 

for an exactresuit, wemust examine f\x{£, rj), y(|, 77 )] |J(§, r/)|, Lhe inLegrand in (6.45). 
Since | J(f, rj) | is biquadratic, and f(x, y) is cubic in x, the integrand is a polynomial of 
degree 5 in x. Therefore, third-order integration will suffice. The program used for the 
computations is similar to the one in Example 6.15: 

#!/usr/bin/python 
## example6_16a 
from gaussQuad2 import * 
from numarray import array 
from math import sqrt 

def f(x,y): 

return (x**2 + y**2)/2.0 \ 

- (x**3 - 3.0*x*y**2)/6.0 \ 

- 2.0/3.0 


x = array([-1.0,-1.0,2.0,2.0]) 
y = arrayC[sqrt(3.0),-sqrt(3.0),0.0,0.0]) 
m = eval(raw_input(’’Integration order ==> ’’)) 
print ’’Integral gaussQuad2(f,x,y,m) 

raw_input(’’ \nPress return to exit’’) 

Elere is the output: 

Integration order ==> 3 
Integral = -1.55884572681 

Solution of Part (2) The following program utilizes triangleQuad: 

#!/usr/bin/python 
# example6_16b 

19 This function is identical to the Prandtl stress function for torsion of a bar with the cross section 
shown; the integral is related to the torsional stiffness of the bar. See, for example Timoshenko, 
S. P., and Goodier, J. N., Theory ofElasticity, 3rd ed., McGraw-Hill, 1970. 
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from numarray import array 
from math import sqrt 
from triangleQuad import * 

def f(x,y): 

return (x**2 + y**2)/2.0 \ 

- (x**3 - 3.0*x*y**2)/6.0 \ 

- 2.0/3.0 

xCorner = array([-1.0, -1.0, 2.0]) 

yCorner = array([sqrt(3.0), -sqrt(3.0), 0.0]) 

print ’’Integral =’’,triangleQuad(f,xCorner,yCorner) 

raw_input(’’Press return to exit’’) 

Since the integrand is a cubic, this quadrature is also exact, the resuit being 

Integral = -1.55884572681 

Note that only four function evaluations were required when using the triangle 
formulas. In contrast, the function had to be evaluated at nine points in part (1). 

EXAMPLE 6.17 

The corner coordinates of a triangle are (0,0), (16,10) and (12,20). Compute 
f f A (x 2 — y 2 ) dx dy over this triangle. 

Solution 



Because /(x, y) is quadratic, quadrature over the three integration points shown 
in Fig. 6.10(b) will be sufficient for an “exact” resuit. Note that the integration points 
lie in the middle of each side; their coordinates are (6,10), (8, 5) and (14,15). The area 
of the triangle is obtained from Eq. (6.49): 


1 

1 1 1 

i 

1 1 1 

/1= - 
2 

Xi x 2 x 3 

yi yi yi 

~ 2 

0 16 12 

0 10 20 


= 100 
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From Eq. (6.51) we get 

C 

I=aJ 2 w k f{x k , y k ) 


k=a 

= 100 

= ^ [(6 2 - 10 2 ) + (8 2 - 5 2 ) + (14 2 - 15 2 )] = 1800 


|/(6, 10) + | /(8, 5) + i /(14, 15) 


PROBLEM SET6.3 

1. Use Gauss-Legendre quadrature to compute 



(1 — x 2 )(l — y 2 ) dxdy 


2. Evaluate the following integral with Gauss-Legendre quadrature: 

r 2 p3 


’y =' 

3. Compute the approximate value of 

-i r \ 


/ / x 2 y 2 dxdy 

J y=0 J x=0 


J J e (J,2+y2) dx dy 


with Gauss-Legendre quadrature. Use integration order (a) two and (b) three. 
(The “exact” value of the integral is 2.230 985.) 

4. Use third-order Gauss-Legendre quadrature to obtain an approximate value of 



jt{x — y) 
2 


dxdy 


(The “exact” value of the integral is 1.621139.) 

5. 



Map the integral / f A xy dxdy from the quadrilateral region shown to the “Stan¬ 
dard” rectangle and then evaluate it analytically. 
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6 . 



Compute / J A xdxdy over the quadrilateral region shown by first mapping it into 
the “Standard” rectangle and then integrating analytically. 



Use quadrature to compute / J A x 2 dx dy over the triangle shown. 

8 . Evaluate / f A x 3 dx dy over the triangle shown in Prob. 7. 

9. 



Evaluate / J A [3 — x) y dx dy over the region shown. 

10. Evaluate / J A x 2 ydxdy over the triangle shown in Prob. 9. 

11 . ■ 



Evaluate f f A xy{2 — x 2 ) (2 — xy) dx dy over the region shown. 

12. ■ Compute / / A xyexp(— x 2 ) dxdy over the region shown in Prob. 11 tofourdec- 
imal places. 
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13. ■ 



Evaluate / f A (1 — x) (y — x) y dx dy over the triangle shown. 

14. ■ Estimate / f A sin jtx dx dy over the region shown in Prob. 13. Use the cubic 
integration formula for a triangle. (The exact integral is 1/jr.) 

15. ■ Compute f f A smjTXsmjT(y — x) dxdy to six decimal places, where A is 
the triangular region shown in Prob. 13. Consider the triangle as a degenerate 
quadrilateral. 



Write a program to evaluate / f A /(x, y) dx dy over an irregular region that has 
been divided into several triangular elements. Use the program to compute 
/ Ia x y(y — x) dxdy over the region shown. 











Initial Value Problems 


Solve y' = F(x, y) with the auxiliary conditions y (a) = a 


7.1 Introduction 

The general form of a first-order differential equation is 

/=/(x, y) (7.1 a) 

where/ = dy/dx and /(x, y) is a givenfunction. The solutionofthis equation contains 
an arbitrary constant (the constant of integratio n). To find this constant, we must know 
a point on the solution curve; that is, y must be specified at some value of x, say at 


x = a. We write this auxiliary condition as 

y{a) = a (7.1b) 

An ordinary differential equation of order n 

y [n) = /(x,y,y'"- 11 ) (7.2) 

can always transformed into n first-order equations. Using the notation 

yo = y yi = / y> = y" • ■ • y n -\ = y (n_1) (7.3) 

the equivalent first-order equations are 

/o = yi y = yz y 2 = ys y„= /(X, y 0 ,yi,...,y„_i) (7.4a) 


The solution now requires the knowledge n auxiliary conditions. If these conditions 
are specified at the same value of x, the problem is said to be an initial value problem. 
Then the auxiliary conditions, called initial conditions, have the form 

yb(a)=ao yda) =ai ... y n -i[a) = a n -i (7.4b) 
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7.2 Taylor Series Method 


If yt are specified at different values of x, the problem is called a boundary value 
problem. 

For example, 

/ = -y y( 0 ) = i yco) = o 

is an initial value problem since both auxiliary conditions imposed on the solution 
are given at x = 0. On the other hand, 

/ = -y y(0) = i y(TT) = o 

is a boundary value problem because the two conditions are specified at different 
values of x. 

In this chapter we consider only initial value problems. The more difficult bound¬ 
ary value problems are discussed in the next chapter. We also make extensive use of 
vector notation, which allows us manipulate sets of first-order equations in a concise 
form. For example, Eqs. (7.4) are written as 


where 


y 1 = F(x, y) 


y(a) = a 


F {x, y) 


34 

T2 


. /(*. y) _ 


(7.5a) 


(7.5b) 


A numerical solution of differential equations is essentially a table of x- and y-values 
listed at discrete intervals of x. 


7.2 Taylor Series Method 

The Taylor series method is conceptually simple and capable of high accuracy. Its 
basis is the truncated Taylor series for y about x: 

y(x + h ) «y(x) +y '{x)h+ ^y"(x )/f + ^/"(xjfz 3 4-F -^y M [x')h m (7.6) 

Because Eq. (7.6) predicts y at x + h from the information available at x, it is also a 
formula for numerical integration. The last term kept in the series determines the 
order of integration. For the series in Eq. (7.6) the integration order is m. 

The truncation error, due to the terms omitted from the series, is 

E =---y (mfl) (§)h m+1 , x < § < x + h 

[m+ l)r 5 s 
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Using the finite difference approximation 

v (,m-i) fn ~ y (m) (x + h) - y (m) (x) 

y ~ h 

we obtain the more usable form 

h m 

E » --— [y w (x + h) - (x)] (7.7) 

[m+ 1 )! 

which could be incorporated in the algorithm to monitor the error in each integration 
step. 

■ taylor 

The function t aylor implements the Taylor series method of integration of order four. 
It can handle any number of first-order differential equations j/ = //(x, y () , y lt ...), 
i = 0,1,.... The user is required to supply the function deriv that returns the 4 x n 
array 


' (yO r " 


y 0 

/i • 

• y n -x 

[y") T 


/o' 

// ■ 

■ Vn -1 

(y"') r 


/o" 

y/' ■ 

• fl 1 

_(y (4) ) r _ 


k 1 

y{ 4) ■ 

• y (4) i 


The function returns the arrays X and Y that contain the values of x and y at intervals h. 
## module taylor 

’’’ X,Y = taylor(deriv,x,y,xStop,h). 

4th-order Taylor series method for solving the initial 

value problem {y>’ = {F(x,{y>)>, where 

{ Y > = {y[0],y[1],...y[n-l]>. 

x,y = initial conditions 

xStop = terminal value of x 

h = increment of x used in integration 

deriv = user-supplied function that returns the 4 x n array 
[y ’[o] y’ [1] y’[2] ... y’[n-i] 

y’ ’ [0] y ” [1] y”[2] ... y’ ’ [n-l] 

y’ ’ ’ [0] y” ’ [1] y’ ’ ’ [2] ... y” ’ [n-l] 
y-[0] y’ ’ ’ ’ [1] y-[2] . . . y’ ” ’ [n-l]] 

from numarray import array 
def taylor(deriv,x,y,xStop,h): 

X = [] 
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7.2 Taylor Series Method 


Y = [] 

X. append(x) 

Y. append(y) 
while x < xStop: 


# Loop over integration steps 


h = min(h,xStop - x) 


D = deriv(x,y) 

H = 1.0 

for j in range(4): 


# Derivatives of y 


# Build Taylor series 


H = H*h/(j + 1) 
Y = Y + D[j]*H 


# H = h" j/j ! 


x = x + h 

X. append(x) 

Y. append(y) 


# Append results to 

# lists X and Y 

# Convert lists into arrays 


return array(X),array(Y) 


■ printSoln 

We use this function to print X and Y obtained from numerical integration. The amount 
of data is controlled by the parameter freq. For example, if freq = 5, every 5th 
integration step would be displayed. If f r eq = 0 , only the initial and final values will 
be shown. 

## module printSoln 
’’’ printSoln(X,Y,freq) . 

Prints X and Y returned from the differential 
equation solvers using printput frequency ’freq’. 
freq = n prints every nth step. 
freq = 0 prints initial and final values only. 

def printSoln(X,Y,freq): 

def printHead(n): 


print '’\n 


x 


for i in range (n): 


print ’’ y[’’ i i,’’] 

print 


def printLine(x,y,n): 

print ’’%13.4e’’% x, 
for i in range (n): 
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print ’’%13.4e’’% y[i], 
print 

m = len(Y) 

try: n = len(Y[0]) 

except TypeError: n = 1 

if freq == 0: freq = m 

printHead(n) 

for i in range(0,m,freq): 
printLine(X[i],Y[i],n) 

if i != m - 1: printLine(X[m - l],Y[m - l],n) 

EXAMPLE 7.1 

Given that 


/ + 4y = x 2 y(0) = l 


determine y(0.1) with the fourth-order Taylor series method using a single integration 
step. Also compute the estimated error from Eq. (7.7) and compare it with the actual 
error. The analytical solution of the differential equation is 


31 -4x 1 2 1 1 

y = —e H—x- x-t - 

J 32 4 8 32 


Solution The Taylor series up to and including the term with h 4 is 

y{h) = y(0) + /(0) h + ^ /(0) h 2 + ^ /'(0) h 3 + ^ j/ 4) (0) h 4 (a) 

Differentiation of the differential equation yields 
y = —4y + y 2 

y' = -4y + 2x= 16y - 4x 2 + 2x 

= 16 y — 8x + 2 = —64y + 16X 2 — 8x + 2 
y {4) = -64/ + 32x - 8 = 256y - 64X 2 + 32x - 8 


Thus at x = 0 we have 


/(0) = -4(1) = -4 
/'(0) = 16(1) = 16 
/"(0) = -64(1) + 2 = -62 
y (4) (0) = 256(1) - 8 = 248 
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With h = 0.1 Eq. (a) becomes 

y(0.1) = 1 + (-4) (0.1) + I(16)(0.1) 2 + i(-62)(0.1) 3 + i(248)(0.1) 4 
= 0.670700 

According to Eq. (7.7) the approximate truncation error is 

E= ^[/ 4) (°.i)-y (4) (0)] 

where 

y (4) (0) = 248 

y (4) (0.1) = 256(0.6707) - 64(0.1) 2 + 32(0.1) - 8 = 166.259 

Therefore, 

(0 l) 4 

E = (166.259 - 248) = -6.8 x 10“ 5 

The analytical solution yields 

y(0.1) = ^e- 4(01) + -Vl) 2 - 4(0.1) + 4- = 0.670623 

oZ 4 o oZ 

so that the actual error is 0.670623 — 0.670700 = —7.7 x 10 -5 . 


EXAMPLE 7.2 

Solve 


y" = - 0 .I 3 / - x y( 0 ) = 0 /( 0 ) = 1 

from x = 0 to 2 with the Taylor series method of order four. Use h = 0.25 and utilize 
the functions taylor and printSoln. 


Solution With the notation y 0 = y and yi = / the equivalent first-order equations 
and the initial conditions are 


/o 


yi 

_/l_ 


i 

1 

o 

i—• 

1 

X 

i_ 


y( 0 ) = 


o 

i 


Repeated differentiation of the differential equations yields 


y, 


—O.lyi — x 

.-o.i y-i_ 


O.Olyi + O.lx — 1 


'-o.iyi - 1 ' 


O.Olyi + O.lx - 1 

o.oiy + o.i 


-O.OOlyi - O.Olx+O.l 


o.oiy + 0.1 
-o.ooiy - o.oi 


-O.OOlyi - O.Olx+O.l 
O.OOOlyi + O.OOlx — 0.01 























254 


Initial Value Problems 


Thus the derivative array that has to be computed by the function deriv is 

—O.ljy - x 
O.Olyi + O.lx - 1 
- 0 . 001)4 - 0.01x + 0.1 
0 .0001)4 + O.OOlx — 0.01_ 

Here is the program that performs the integration: 

#!/usr/bin/python 
## example7_2 
from printSoln import * 
from taylor import * 


D 


—0.1)4 — x 
O.Olyi + O.lx - 1 
-0.001)4 - O.Olx + O.l 


def deriv(x,y): 


D = 
D[0] 
D[l] 
D[2] 
D[ 3] 


zeros((4,2),type=Float64) 

= [y[1] , -0.l*y[l] - x] 


= [D[0,1], 0.01*y[l] + 0.1-x - 1.0] 

= [D[1,1], -0.001*y[l] - 0.01*x + 0.1] 

= [D[2,1], 0.0001*y[l] + 0.001*x - 0.01] 


return D 


x = 0.0 # Start of integration 

xStop =2.0 # End of integration 

y = array([0.0, 1.0]) # Initial values of {y} 
h = 0.25 # Step size 

freq =1 # Printout frequency 

X,Y = taylor(deriv,x,y,xStop,h) 
printSoln(X,Y,freq) 

raw_input(’’ \nPress return to exit’’) 


The results are: 


X 

y[ o ] 

y[ 1 ] 

0.0000e+000 

0.0000e+000 

1.0000e+000 

2.5000e-001 

2.4431e-001 

9.4432e-001 

5.0000e-001 

4.6713e-001 

8.2829e-001 

7.5000e-001 

6.5355e-001 

6.5339e-001 

1.0000e+000 

7.8904e-001 

4.2110e-001 

1.2 500e+000 

8.5943e-001 

1.3281e-001 

1.5000e+000 

8.5090e-001 

-2.1009e-001 

1.7500e+000 

7.499 5e-001 

-6.0625e-001 

2.0000e+000 

5.4345e-001 

-1.0543e+000 
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The analytical solution of the problem is 

y = lOOx - 5x 2 + 990(e-° lj: - 1) 

from which we obtain y(2) = 0.543 446, which agrees well with the numerical solution. 


7.3 Runge-Kutta Methods 

The main drawback of the Taylor series method is that it requires repeated differen- 
tiation of the dependent variables. These expressions may become very long and are, 
therefore, error-prone and tedious to compute. Moreover, there is the extra work of 
coding each of the derivatives. The aim of Runge-Kutta methods is to eliminate the 
need for repeated differentiation of the differential equations. Since no such differen- 
tiation is involved in the first-order Taylor series integration formula 

y(x + h) = y(x) + /(x) h = y(x) + F(x, y) h (7.8) 

it can also be considered as the first-order Runge-Kutta method; it is also called Euler’s 
method. Due to excessive truncation error, this method is rarely used in practice. 



Figure 7.1. Graphical representation of Euler’s formula. 


Let us now take a look at the graphical interpretation of Euler’s formula. For the 
sake of simplicity, we assume that there is a single dependent variable y, so that 
the differential equation is / = /(x, y). The change in the solution y between x and 
X+ h is 

px+h px+h 

y{x + h) — y(h) = y dx= I /(x, y)dx 

J X J X 

which is the area of the panel under the /(x) plot, shown in Fig. 7.1. Euler’s formula 
approximates this area by the area of the cross-hatched rectangle. The area between 
the rectangle and the plot represents the truncation error. Clearly, the truncation error 
is proportional to the slope of the plot; that is, proportional to /'(x). 

Second-Order Runge-Kutta Method 

To arrive at the second-order method, we assume an integration formula of the form 
y(x + h) = y(x) + c 0 F(x, y )h + CiF [x + ph. y + qh F(x, y)] h (a) 
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and attempt to find the parameters c 0 , c t , p and q by matching Eq. (a) to the Taylor 
series 


y(x + h) 


y(x) + y{x)h+ + Offl 3 ) 

y(x) + F(x, y )h+ ^F'(x, y)lf + 0{tf) 


Noting that 


F' (x, y) 


9F p 9F , 9F p, 9F ^ 

-1- > —Vi =-1- > —F,(x, y) 

a* U d y* dx 


(b) 


where «is the number of first-order equations, we can write Eq. (b) as 


y(x + h) = y(x)+F(x,y )h + \ ^ ^F,-(x, $) ^ + CKfc 3 ) (c) 

1 \° X 1=0 d 3 / ! / 

Returning to Eq. (a), we can rewrite the last term by applying a Taylor series in 
several variables: 


9F 0p 

F [x + ph, y + qh¥{x, y)] = F(x, y) + — ph+ qh^ — F,(x, y) + CK/z 2 ) 

9x !=1 9y,- 


so that Eq. (a) becomes 
y(x + h) = y(x) + (c 0 + cj F(x, y)/z+ Ci 


3F i Qp 

-ph+qh^—Fdx.y) 


dX 


h+Oitf) (d) 


Comparing Eqs. (c) and (d), we find that they are identical if 

1 1 

c 0 + ci = l c\p = — C\q = - (e) 

Because Eqs. (e) represent three equations in four unknown parameters, we can assign 
any value to one of the parameters. Some of the popular choices and the names 
associated with the resulting formulas are: 


Co 

= 0 

C\ 

= 1 

P— 1/2 

q = 

1/2 

Modifled Euler’s method 

Co 

= 1/2 

Cl 

= 1/2 

p= 1 

q = 

1 

Heun’s method 

Co 

= 1/3 

Ci 

= 2/3 

P= 3/4 

q = 

3/4 

Ralston's method 


Ali these formulas are classified as second-order Runge-Kutta methods, with no for¬ 
mula having a numerical superiority over the others. Choosing the modifled Euler’s 
method, we substitute the corresponding parameters into Eq. (a) to yield 


y(x + h ) = y(x) + F 


h h 

x+ 2’ y + 2 F M 


(f) 
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This integration formula can be conveniently evaluated by the following sequence of 
operatioris 

K 0 = hF(x, y) 

K x = «Ff*+|.y+^KoJ (7-9) 

y(x + h) = y(x) + Ki 

Second-order methods are not popular in computer application. Most programmers 
prefer integration formulas of order four, which achieve a given accuracy with less 
computational effort. 


y'(x) 



Figure 7.2. Graphical representation of modified Euler 
formula. 


Figure 7.2 displays the graphical interpretation of modified Euler’s formula for 
a single differential equation / = f[x, y). The first of Eqs. (7.9) yields an estimate of 
y at the midpoint of the panel by Euler’s formula: y{x + h/2) = y(x) + /(x, y) h/2 = 
y(x) + K 0 /2. The second equation then approximates the area of the panel by the area 
K\ of the cross-hatched rectangle. The error here is proportional to the curvature /" 
of the plot. 


Fourth-Order Runge-Kutta Method 


The fourth-order Runge-Kutta method is obtained from the Taylor series along the 
same lines as the second-order method. Since the derivation is rather long and not very 
instructive, we skip it. The final form of the integration formula again depends on the 
choice of the parameters; that is, there is no unique Runge-Kutta fourth-order formula. 
The most popular version, which is known simply as the Runge-Kutta method, entails 
the following sequence of operations: 


K 0 

Ki 

K 2 

k 3 

y(x + h) 


hF{x, y) 

, / h 
W ( x + 2 ,y + 

hF I x + -,y + 



h F(x + h, y+ K 2 ) 

yCxl + ^CKo + ZKr+ZKz + Ks) 
6 


(7.10) 
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The main drawback of this method is that it does not lend itself to an estimate of the 
truncation error. Therefore, we must guess the integration step size h, or determine 
it by trial and error. In contrast, the so-called adaptive methods can evaluate the 
truncation error in each integration step and adjust the value of h accordingly (but at 
a higher cost of computation). One such adaptive method is introduced in the next 
article. 

■ run_kut4 

The function integrate in this module implements the Runge-Kutta method of 
order four. The user must provide integrate with the function F(x, y) that defines 
the first-order differential equations y 7 = F(jc, y). 

## module run_kut4 

’’’ X,Y = integrate(F,x,y,xStop,h). 

4th-order Runge-Kutta method for solving the 

initial value problem {y}’ = {F(x,{y>)>, where 

{y} = fy[0],y[i],...y[n-i]}. 

x,y = initial conditions. 

xStop = terminal value of x. 

h = increment of x used in integration. 

F = user-supplied function that returns the 

array F(x,y) = {y’[0],y’[1],...,y’[n-1]}. 

from numarray import array 
def integrate(F,x,y,xStop,h): 

def run_kut4(F,x,y,h): 

# Computes increment of y from Eqs. (7.10) 

K0 = h*F(x,y) 

K1 = h*F(x + h/2.0, y + K0/2.0) 

K2 = h*F(x + h/2.0, y + Kl/2.0) 

K3 = h*F(x + h, y + K2) 

return (K0 + 2.0*K1 + 2.0*K2 + K3)/6.0 

X = [] 

Y = [] 

X. append(x) 

Y. append(y) 
while x < xStop: 

h = min(h,xStop - x) 
y = y + run_kut4(F,x,y,h) 
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x = x + h 

X. append(x) 

Y. append(y) 

return array(X),array(Y) 

EXAMPLE 7.3 

Use the second-order Runge-Kutta method to integrate 

Y = sin y y(0) = 1 

from x = 0 to 0.5 in steps of h = 0.1. Keep four decimal places in the computations. 
Solution In this problem we have 

F ( x , y ) = sin y 

so that the integration formulas in Eqs. (7.9) are 
K 0 = hF[x, y) = 0.1 sin y 

K\ = hF y+ = 0.1 sin ^y + ^K 0 j 

y(x + h ) = y[x) + Ki 

Noting that y(0) = 1, we may proceed with the integration as follows: 

K 0 = 0.1 sin 1.0000 = 0.0841 

/ 0.0841 \ 

K x = 0.1 sin I 1.0000 H --— J = 0.0863 

y(0.1) = 1.0 + 0.0863 = 1.0863 

K 0 = 0.1 sin 1.0863 = 0.0885 

/ 0.0885 \ 

K x = 0.1 sin I 1.0863 + —— j = 0.0905 
y(0.2) = 1.0863 + 0.0905 = 1.1768 

and so on. A summary of the computations is shown in the table below. 


X 

y 

K 0 

Ki 

0.0 

1.0000 

0.0841 

0.0863 

0.1 

1.0863 

0.0885 

0.0905 

0.2 

1.1768 

0.0923 

0.0940 

0.3 

1.2708 

0.0955 

0.0968 

0.4 

1.3676 

0.0979 

0.0988 

0.5 

1.4664 
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The exact solution can be shown to be 

x(y) = ln(csc y - cot y) + 0.604582 

whichyields x(l.4664) = 0.5000. Therefore, up to this point the numerical solution is 
accurate to four decimal places. However, it is unlikely that this precision would be 
maintained if we were to continue the integration. Since the errors (due to truncation 
and roundoff) tend to accumulate, longer integration ranges require better integration 
formulas and more significant figures in the computations. 

EXAMPLE 7.4 

Solve 


y" = — 0 . 1 / — x y( 0 ) = 0 3 /( 0 ) = 1 


from x = 0 to 2 in increments of h = 0.25 with the Runge-Kutta method of order four. 
(This problem was solved by the Taylor series method in Example 7.2.) 


Solution Letting y 0 = y and yi = 3 /, we write the equivalent first-order equations as 


y 7 = F(x, y) 


/0 


J4 

_/l_ 


-O.lyi - x 


Comparing the function F(x , y)here with deriv(x , y)in Example 7.2 we note that it 
is much simpler to input the differential equations in the Runge-Kutta method than 
in the Taylor series method. 


#!/usr/bin/python 
## example7_4 

from numarray import array,zeros,Float64 
from printSoln import * 
from run_kut4 import * 


def F(x,y): 

F = zeros((2),type=Float64) 


F[0] = y[1] 

F[1] = -0.1*y[l] 
return F 

x = 0.0 

xStop = 2.0 

y = array([0.0, 1.0]) 

h = 0.25 

freq = 1 


# Start of integration 

# End of integration 

# Initial values of {y} 

# Step size 

# Printout frequency 


x 
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X,Y = integrate(F,x,y,xStop,h) 
printSoln(X,Y,freq) 

raw_input(’’Press return to exit’’) 

The output from the fourth-order method is shown below. The results are the 
same as obtained by the Taylor series method in Example 7.2. This was expected, 
since both methods are of the same order. 


X 

y[ 0 ] 

y[ 1 1 

0.0000e+000 

0.0000e+000 

1.0000e+000 

2.5000e-001 

2.4431e-001 

9.4432e-001 

5.0000e-001 

4.6713e-001 

8.2829e-001 

7.5000e-001 

6.5355e-001 

6.5339e-001 

1.0000e+000 

7.8904e-001 

4.2110e-001 

1.2 500e+000 

8.5943e-001 

1.3281e-001 

1.5000e+000 

8.5090e-001 

-2.1009e-001 

1.7500e+000 

7.499 5e-001 

-6.062 5e-001 

2.0000e+000 

5.4345e-001 

-1.0543e+000 


EXAMPLE 7.5 

Use the fourth-order Runge-Kutta method to integrate 

y = 3y-4e~ x y( 0) = 1 

from x = 0 to 10 in steps of h= 0.1. Compare the resuit with the analytical solution 
y = e~ x . 

Solution We used the program shown below. Recalling that run_kut4 assumes y to 
be an array, vve specified the initial valu e as y = arrayC [l. 0]) ratherthany = l.o. 

#!/usr/bin/python 
## example7_5 

from numarray import zeros,Float64,array 
from run_kut4 import * 
from printSoln import * 
from math import exp 


def F(x,y): 

F = zeros((1),type=Float64) 

F[0] = 3.0*y[0] - 4.0*exp(-x) 
return F 
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x = 0.0 
xStop = 10.0 
y = array([1.0]) 
h = 0.1 
freq = 20 


# Start of integration 

# End of integration 

# Initial values of {y} 

# Step size 

# Printout frequency 


X,Y = integrate(F,x,y,xStop,h) 
printSoln(X,Y,freq) 

raw_input(’’ \nPress return to exit’’) 


Running the program produced the following output: 


x 

0.0000e+000 
2.0000e+000 
4.0000e+000 
6.0000e+000 
8.0000e+000 
1.0000e+001 


y[ o ] 

1.OOOOe+OOO 
1.32 50e-001 
-1.1237e+000 
-4.60 56e+002 
-1.8 5 75e+00 5 
- 7.4912e+00 7 


It is ciear that something went wrong. According to the analytical solution, y 
should approach zero with increasing x, but the output shows the opposite trend: 
after an initial decrease, the magnitude of y increases dramatically. The explanation 
is found by taking a closer look at the analytical solution. The general solution of the 
given differential equation is 


y = Ce 3 * + e~ x 

which can be verified by substitution. The initial condition y(0) = 1 yields C = 0, so 
that the solution to the problem is indeed y = e~ x . 

The cause of trouble in the numerical solution is the dormant term Ce 3 *. Suppose 
that the initial condition contains a small error e, so that we have y(0) = 1 + e. This 
changes the analytical solution to 

y = e e 3 * + e~ x 

We now see that the term containing the error e becomes dominant as x is increased. 
Since errors inherent in the numerical solution have the same effect as small changes in 
initial conditions, we conclude that our numerical solution is the victim of numerical 
instability due to sensitivity of the solution to initial conditions. The lesson is: do not 
blindly trust the results of numerical integration. 
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EXAMPLE 7.6 



\|Vh 


A spacecraft is launched at altitude H = 772 km above sea level with the speed 
v 0 = 6700 m/s in the direction shown. The differential equations describing the mo- 
tion of the spacecraft are 


.. a2 GMe 

r = r9 -— 


9 = -- 


2 r9 


where r and 6 are the polar coordinates of the spacecraft. The constants involved in 
the motion are 


G = 6.672 x 10 _u m 3 kg - 1 s -2 = universal gravitational constant 
Me = 5.9742 x 10 24 kg = mass of the earth 
Re = 6378.14 km = radius of the earth at sea level 


(1) Derive the first-order differential equations and the initial conditions of the form 
y = F(f, y), y(0) = b. (2) Use the fourth-order Runge-Kutta method to integrate the 
equations from the time of launch until the spacecraft hits the earth. Determine 6 at 
the impact site. 

Solution of Part (1) We have 

GMe = (6.672 x 10 -11 ) (5.9742 x 10 24 ) = 3.9860 x 10 14 mV 2 


Letting 


"To" 


r 

yi 


r 

T 2 


e 

_T3. 


_e _ 


the equivalent first-order equations become 


"yo" 


yi 

u 


y 0 j! - 3.9860 x 10 14 /yj 

y2 


ys 

_y3_ 


— 2 yij^/yo 
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and the initial conditions are 

r(0) = Re + H = Re = (6378.14 + 772) x 10 3 = 7.15014 x 10 6 m 
r(0) = 0 
0 ( 0 ) = 0 

0(0) = Vo/rm = (6700) /(7.15014 x 10 6 ) = 0.937045 x 10“ 3 rad/s 

7.15014 x 10 6 
0 
0 

0.937045 x 10“ 3 _ 

Solution of Part (2) The program used for numerical integration is listed below. 
Note that the independent variable t is denoted by x. The period of integration 
xStop (the time when the spacecraft hits) was estimated from a previous run of the 
program. 

#!/usr/bin/python 
## example7_6 

from numarray import zeros,Float64,array 
from run_kut4 import * 
from printSoln import * 


Therefore, 


y(0) 


def F(x,y): 


F = zeros((4),type=Float64) 

F[0] = y[1] 

F[1] = y[0]*(y[3]**2) - 3.9860el4/(y[0] 
F[2] = y[3] 

F[3] = -2.0*y[l]*y[3]/y[0] 
return F 


* * 2 ) 


x = 0.0 

xStop = 1200.0 
y = array([7.15014e6, 
h = 50.0 
freq = 2 


0.0, 0.0, 0.937045e-3]) 


X,Y = integrate(F,x,y,xStop,h) 
printSoln(X,Y,freq) 

raw_input(’’ \nPress return to exit’’) 
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Here is the output: 


x 

0.0000e+000 
1.0000e+002 
2.0000e+002 
3.0000e+002 
4.0000e+002 
5.0000e+002 
6.0000e+002 
7.0000e+002 
8.0000e+002 
9.0000e+002 
1.0000e+003 
1.1000e+003 
1.2000e+003 


y[ 0 ] 

7.1501e+006 
7.1426e+006 
7.1198e+006 
7.0820e+006 
7.0294e+006 
6.9622e+006 
6.8808e+006 
6.78 56e+006 
6.6773e+006 
6.5568e+006 
6.42 50e+006 
6.2831e+006 
6.1329e+006 


y[ 1 ] 

0.0000e+000 
-1.5173e+002 
-3.02 76e+002 
-4.5236e+002 
- 5.99 73e+002 
-7.4393e+002 
-8.8389e+002 
-1.0183e+003 
-1.1456e+003 
-1.2639e+003 
-1.3708e+003 
-1.4634e+003 
-1.5384e+003 


y[ 2 ] 

0.0000e+000 
9.3771e-002 

1.8 794e-001 
2.8292e-001 
3.7911e-001 
4.769 7e-001 
5.7693e-001 
6.7950e-001 

7.8 520e-001 
8.9459e-001 
1.0083e+000 
1.1269e+000 
1.2512e+000 


y[ 3 ] 

9.3704e-004 
9.3904e-004 
9.4504e-004 
9.5515e-004 
9.6951e-004 
9.8832e-004 
1.0118e-003 
1.0404e-003 
1.0744e-003 
1.1143e-003 
1.1605e-003 
1.213 5e-003 
1.2737e-003 


The spacecraft hits the earth when r equals A’ e = 6.378 14 x 10 6 m. This occurs 
between t = 1000 and 1100 s. Amore accurate value of tcan be obtainedbypolynomial 
interpolation. If no great precision is needed, linear interpolation will do. Letting 
1000 + A t be the time of impact, we can write 

r(1000 + At) = Re 

Expanding r in a two-term Taylor series, we get 

r(1000) + r(1000) At = Re 

6.4250 x 10 6 + (-1.3708 x 10 3 ) At = 6378.14 x 10 3 


from which 


At = 34.184 s 


Thus the time of impact is 1034.25. 

The coordinate 0 of the impact site can be estimated in a similar manner. Using 
again two terms of the Taylor series, we have 

0(1000 + At) = 0(1000) + 0(1000) At 

= 1.0083 + (1.1605 x 10“ 3 ) (34.184) 


= 1.0480 rad = 60.00 
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PROBLEM SET 7.1 

1. Given 

y + 4y = x 2 y(0) = 1 

compute y(0.1) usingtwo steps of theTaylor series method of order two. Compare 
the resuit with Example 7.1. 

2. Solve Prob. 1 with one step of the Runge-Kutta method of order (a) two and (b) four. 

3. Integrate 

y = sin y y( 0 ) = 1 


fromx = 0 to 0.5 with the second-order Taylor series method using /i = 0.1. Com¬ 
pare the resuit with Example 7.3. 

4. Verify that the problem 

y = y 1 / 3 y(0) = 0 

has two Solutions: y = 0 and y = (2 jc/ 3) 3/2 . Which of the Solutions would be re- 
produced by numerical integration if the initial condition is set at (a) y = 0 and 
(b) y = 10 -16 ? Verify your conclusions by integrating with any numerical method. 

5. Convert the following differential equations into first-order equations of the form 
y' = F(x, y): 

(a) ln y + y = sin x 

(b) /y-xj/ -2y 2 = 0 

(c) y t4) - 4/V1 — y 2 = 0 

(d) (y ") 2 = |32/x— f\ 

6 . In the following sets of coupled differential equations t is the independent vari- 
able. Convert these equations into first-order equations of the form y = F(f, y): 


(a) y = x — 2 y x = y — x 

(b) y = -y(j > 2 + i 2 ) 1/4 x = —x (y 2 + x ) 1/4 - 32 

(c) y 2 + rsiny = 4x xx+tcosy=4y 


7. ■ The differential equation for the motion of a simple pendulum is 


where 


tf8 

~dfi 


— — sin 6 


6 = angular displacement from the vertical 
g = gravitational acceleration 
L = length of the pendulum 
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With the transformation r = tjg/L the equation becomes 


cfe 

dc* 


— sin# 


Use numerical integration to determine the period of the pendulum if the ampli - 
tude is #o = 1 rad. Note that for small amplitudes (sin 9 ~ 9) the period is 2tc , jL/g. 

8. ■ A skydiver of mass mina vertical free fall experiences an aerodynamic drag 
force / ; d = Cd V 2 , where y is measured downward from the start of the fall. The 
differential equation describing the fall is 


Determine the time of a 500 m fall. Use g = 9.80665 m/s 2 , c D = 0.2028 kg/m and 
m= 80 kg. 


9. ■ 



The spring-mass system is at rest when the force P(t) is applied, where 

10tN when t < 2 s 
20 N when t> 2 s 

The differential equation of the ensuing motion is 

P{t) k 

y= - y 

m m 

Determine the maximum displacement of the mass. Use m= 2.5 kg and k = 
75 N/m. 

10. ■ 


Water 


y 




The conical float is free to slide on a vertical rod. When the float is disturbed 
from its equilibrium position, it undergoes oscillating motion described by the 
differential equation 


y=g(l-a/) 
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where a = 16 m -3 (determined by the density and dimensions of the float) and 
g = 9.80665 m/s 2 . If the float is raised to the position y = 0.1 m and released, 
determine the period and the amplitude of the oscillations. 



The pendulum is suspended from a sliding collar. The system is at rest when the 
oscillating motion y( t) = Y sin o>L is imposed on the collar, starting at t = 0. The 
differential equation describing the motion of the pendulum is 

g a> 2 

9 = — — sinS + — Ycos9 sin cat 

Plot 9 vs. t from t = 0 to 10 s and determine the largest 9 during this period. Use 
g= 9.80665 m/s 2 , L = 1.0 m, Y = 0.25 m and a> = 2.5rad/s. 



The system consisting of a sliding mass and a guide rod is at rest with the mass 
at r = 0.75 m. At time t = 0 a motor is turned on that imposes the motion 9{t) = 
[jt /12) cos n t on the rod. The differential equation describing the resulting motion 
of the slider is 

r=(—j r sin 2 nt — gsin cos jrf) 

Determine the time when the slider reaches the tip of the rod. Use g = 
9.80665 m/s 2 . 


13 . 
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Aball of mass m= 0.25 kg is launched with the velocity v 0 = 50 m/s in the direc- 
tion shown. If the aerodynamic drag force acting on the ball is / ; d = Cd n 3/2 , the 
differential equations describing the motion are 

Qd .1/2 Cd . 1 /2 

x =- xv ' v= - yv — g 

m m 

where v = v /i 2 + y 2 . Determine the time of flight and the range R. Use Cd = 
0.03 kg/(m-s) 1 / 2 and g = 9.80665 m/s 2 . 

14. ■ The differential equation describing the angular position 9 of a mechanical arm 
is 

.. a{b-6)-66 2 

9 = — -^- 

1 + 9 2 

where a = 100 s -2 and b = 15. If 0(0) = 2n and 6>(0) = 0, compute 9 and 9 when 
t= 0.5 s. 

15. ■ 



The mass m is suspended from an elastic cord with an extensional stiffness k and 
undeformed length L. If the mass is released from rest at 9 = 60° with the cord 
unstretched, find the length r of the cord when the position 9 = 0 is reached for 
the first time. The differential equations describing the motion are 

2 lc 

r = r9 +gcos0 - [r — L) 

m 

.. —2r9 - gsin# 

r 

Use g = 9.80665 m/s 2 , k = 40 N/m, L = 0.5 m and m= 0.25 kg. 

16. ■ Solve Prob. 15 if the mass is released from the position 9 = 60° with the cord 
stretched by 0.075 m. 



y 
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Consider the mass-spring system where dry friction is present between the block 
and the horizontal surface. The frictional force has a constant magnitude /xmg 
(/x is the coefficient of friction) and always opposes the motion. The differential 
equation for the motion of the block can be expressed as 



where y is measured from the position where the spring is unstretched. If the block 
is released from rest at y = y 0) verify by numerical integration that the next positive 
peak value of yis y 0 — 4// mg/k (this relationship can be derived analytically). Use 
k= 3000 N/m, m= 6kg, /x = 0.5, g = 9.80665 m/s 2 and y 0 = 0.1 m. 

18. ■ Integrate the following problems from x = 0 to 20 and plot y vs. x: 


(a) / , + 0.5(y 2 -l)/ + y=0 

(b) /'= ycos2x 


y(0) = 1 /(0) = 0 

y(0) = o y(0) = 1 


These differential equations arise in nonlinear vibration analysis. 

19. ■ The solution of the problem 



is the Bessel function / 0 (x). Use numerical integration to compute / 0 (5) and com¬ 
pare the resuit with —0.17760, the value listed in mathematical tables. Hint to 
avoid singularity at x = 0, start the integration at x = 10 -12 . 

20. ■ Consider the initial value problem 


y"=16.81y y(0) = 1.0 /(0) = -4.1 


(a) Derive the analytical solution. (b) Do you anticipate difficulties in numerical 
solution of this problem? (c) Try numerical integration from x = 0 to 8 to see if 
your concerns were justified. 


21. ■ 



Kirchoffs equations for the Circuit shown are 



dt 


(a) 


— + IU 2 + 2f?(4 + i'i) — E(t) 


(b) 
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where 4 and i 2 are the loop currents, and q 2 is the charge of the condenser. 
Differentiating Eq. (b) and substituting the charge-current relationship dq 2 /dt = 
i 2 , we get 

dii —3 Ri\ — 2 Ri 2 + E(t) 

Tt =-I- (c) 

di 2 2 dii i 2 1 dE , 

— =-1- (d) 

dt 3 dt 3 RC 3 R dt 

We could substitute dii / dt from Eq. (c) into Eq. (d), so that the latter would assume 
theusual form di 2 /dt = f( L, i\ . i 2 ), but it is more convenient toleave the equations 
as they are. Assuming that the voltage source is turned on at time t = 0, plot the 
loop currents 4 and i 2 from t = 0 to 0.05 s. Use E[t) = 240 sin(1207r t) W, R = 1.0 £ 2 , 
L = 0.2 x 10 -3 H and C = 3.5 x IO” 3 F. 


22. ■ 


L L 


- vOOOOO; - 

Q 

-- 

t 

C 1 



' 2 ) 

- mu - 

- mu - 


R R 


The constant voltage source of the Circuit shown is turned on at t = 0, causing 
transient currents 4 and i 2 inthetwoloopsthatlastabout0.05 s. Plot these currents 
fromf = 0 to 0.05 s, usingthefollowingdata: E = 9V, R = 0.25 Q.,L= 1.2 x 10 -3 EI 
and C = 5 x 10 -3 F. Kirchoffs equations for the two loops are 


dii (/ — ch 

t diz i Ri i q2 ~ qi i q2 - n 
L ^ + Rl2 + ^^ + C ~° 


Additional two equations are the current-charge relationships 


dq i . dq 2 

dt ll dt 12 


7.4 Stability and Stiffness 

Loosely speaking, a method of numerical integration is said to be stable if the effects 
of local errors do not accumulate catastrophically; that is, if the global error remains 
bounded. If the method is unstable, the global error will increase exponentially, even- 
tually causing numerical overflow. Stability has nothing to do with accuracy; in fact, 
an inaccurate method can be very stable. 
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Stability is determined by three factors: the differential equations, the method of 
solution and the value of the increment h. Unfortunately, it is not easy to determine 
stability beforehand, unless the differential equation is linear. 

Stability of Euler's Method 

As a simple illustration of stability, consider the linear problem 

y = -ky y(0) = yS (7.11) 

where X is a positive constant. The exact solution of this problem is 

y(x) = /3e~ Xx 

Let us now investigate what happens when we attempt to solve Eq. (7.11) numer- 
ically with Euler’s formula 

y(x + h) = y(x ) + h/(x) (7.12) 

Substituting /(x) = —Xy{x), we get 

y{x + h) = [1 — Xli)y(x) 

If 11 — Xh\ > 1, the method is clearly unstable since \y\ increases in every integration 
step. Thus Euler’s method is stable only if 11 — Xh\ < 1, or 

h< 2/X (7.13) 

The results can be extended to a system of n differential equations of the form 

V = -Ay (7.14) 

where A is a constant matrix with the positive eigenvalues A.,-, 1=1,2,..., n. It can 

be shown that Euler’s method of integration is stable only if 

h< 2/Vax (7.15) 

where X max is the largest eigenvalue of A. 

Stiffness 

An initial value problem is called stiff if some terms in the solution vector y(x) vary 
much more rapidly with x than others. Stiffness can be easily predicted for the differ¬ 
ential equations y = — Ay with constant coefficient matrix A. The solution of these 
equations is y(x) = J2/ C,v, exp(—X/x), where X; are the eigenvalues of A and v, are 
the corresponding eigenvectors. It is evident that the problem is stiff if there is a large 
disparity in the magnitudes of the positive eigenvalues. 
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Numerical integration of stiff equations requires special care. The step size h 
needed for stability is determined by the largest eigenvalue X max , even if the terms 
exp(—>t max x) in the solution decay very rapidly and become insignificant as we move 
away from the origin. 

For example, consider the differential equation 20 

/ + 1001/ + lOOOy = 0 (7.16) 


Using y 0 = y and jy = y 1( the equivalent first-order equations are 


In this case 


? = 


yi 

-1000y 0 - lOOlyi 


A = 


0 

1000 


-1 

1001 


The eigenvalues of A are the roots of 


IA — XI| 


—X -1 

1000 1001 - X 


Expanding the determinant we get 


-M1001 — X) + 1000 = 0 


which has the Solutions A i = 1 and k 2 = 1000. These equation are clearly stiff. Accord- 
ing to Eq. (7.15) we would need h < 2/X 2 = 0.002 for Euler’s method to be stable. The 
Runge-Kutta method would have approximately the same limitation on the step size. 

When the problem is very stiff, the usual methods of solution, such as the Runge- 
Kutta formulas, become impractical due to the very small /zrequired for stability. These 
problems are best solved with methods that are specially designed for stiff equations. 
Stiff problem solvers, which are outside the scope of this text, have much better stabil¬ 
ity characteristics; some of them are even unconditionally stable. However, the higher 
degree of stability comes at a cost—the general rule is that stability can be improved 
only by reducing the order of the method (and thus increasing the truncation error). 

EXAMPLE 7.7 

(1) Show that the problem 

19 

/ = -—y-ioy y(0) = -9 /(0) = 0 


20 This example is taken from C. E. Pearson, Numerical Methods in Engmeering and Science, van 
Nostrand and Reinhold (1986). 
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is moderately stiff and estimate hw.n, the largest value of h for which the Runge-Kutta 
methodwouldbestable. (2) Confirm the estimate bycomputingy(lO) with h ~ !% m x/2 
and h~ 2h max . 


Solution of Part (1) With the notation y = y 0 and / = yi the equivalent first-order 
differential equations are 


V 


1 

G '.1 

t-H 

1 _ 

= -A 

To 

1 

1 

H 

's 

1 

h-* 
O 

1_ 


.■ yi . 


where 


0 -1 



The eigenvalues of A are given by 


1A — XI| 


—k 

19 

T 


-i 

10-A. 


= 0 


which yields /. i = 1/2 and k 2 = 19/2. Because k 2 is quite a bit larger than k i, the 
equations are moderately stiff. 


Solution of Part (2) An estimate for the upper limit of the stable range of h can be 
obtained from Eq. (7.15): 


hin 


ax 


2 

Amax 


2 

1972 


0.2153 


Although this formula is strictly valid for Euler’s method, it is usually not too far off 
for higher-order integration formulas. 

Here are the results from the Runge-Kutta method with h= 0.1 (by specifying 
freq = 0 in printSoln, only the initial and final values were printed): 


x 

0.0000e+000 
1.0000e+001 


y[ 0 ] 

-9.0000e+000 
-6.4011e-002 


y[ 1 ] 

0.0000e+000 
3.2005e-002 


The analytical solution is 

y(x) = ~Y e ~ X/2 + ^e~ Wx/2 


yielding y(10) = —0.0640 11, which agrees with the value obtained numerically. 
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With h = 0.5 we encountered instability, as expected: 


X 


y[ o ] 


y[ i ] 


0.OOOOe+OOO -9.0000e+000 


0.0000e+000 


1.0000e+001 2.7030e+020 -2.5678e+021 


7.5 Adaptive Runge-Kutta Method 

Determination of a suitable step size hcan be a major headache in numerical integra- 
tion. If hi s too large, the truncation error may be unacceptable; if his too small, we are 
squandering computational resources. Moreover, a constant step size may not be ap- 
propriate for the entire range of integration. For example, if the solution curve starts 
off with rapid changes before becoming smooth (as in a stiff problem), we should 
use a small h at the beginning and increase it as we reach the smooth region. This is 
where adaptive methods come in. They estimate the truncation error at each integra¬ 
tion step and automatically adjust the step size to keep the error within prescribed 
limits. 

The adaptive Runge-Kutta methods use so-called embedded integration formu¬ 
las. These formulas come in pairs: one formula has the integration order m, the 
other one is of order m+ 1. The idea is to use both formulas to advance the solution 
from rtor+li. Denoting the results by y m (x + h) and y nn . i (x + h), we may estimate 
the truncation error in the formula of order m: 


E(/i) = Yrru-1 (x+h)- y m (x + h) 


(7.17) 


What makes the embedded formulas attractive is that they share the points where 
F(x, y) is evaluated. This means that once y m (jc + h ) has been computed, relatively 
small additional effort is required to calculate y ?m -i(x + h). 

flere are the Runge-Kutta embedded formulas of orders 5 and 4 that were 
originally derived by Fehlberg; hence they are known as Runge-Kutta-Fehlberg 
formulas: 


Ko = mx, y ) 



(7.18) 


5 


y 5 (x + h) = y(jt) + ^ C/K, (fifth-order formula) 


(7.19a) 


5 


y 4 (x + h) = y(jc) + ^ D,-K,- (fourth-order formula) 


(7.19b) 
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The coefficients appearing in these formulas are not unique. Table 6.1 gives the coef- 
ficients proposed by Cash and Karp 21 which are claimed to be an improvement over 
Fehlberg’s original values. 


i 

Ai 

Bij 

Q 

Di 

0 







37 

2825 







378 

27648 

1 

1 

5 

1 

5 

- 

- 

- 

- 

0 

0 

2 

3 

3 

9 




250 

18 575 


10 

40 

40 




621 

48384 

3 

3 

3 

9 

6 



125 

13 525 

5 

10 

10 

5 



594 

55 296 


1 

11 

5 

70 

35 


0 

277 


54 

2 

27 

27 


14336 

5 

7 

1631 

175 

575 

44275 

253 

512 

1 

8 

55296 

512 

13824 

110592 

4096 

1771 

4 


Table 6.1. Cash-Karp coefficients for Runge-Kutta-Fehlberg formulas 


The solution is advanced with the fifth-order formula in Eq. (7.19a). The fourth- 
order formula is used only implicitly in estimating the truncation error 

5 

E(h)=y 5 (x+h)-y A (x + h) = £(C« - AOK,- (7.20) 

(=0 

Since Eq. (7.20) actually applies to the fourth-order formula, it tends to overestimate 
the error in the fifth-order formula. 

Note that E [h) is a vector, its components F; ( h) representing the errors in the 
dependent variables y,-. This brings up the question: what is the error measure e{h) 
that we wish to control? There is no single choice that works well in all problems. If 
we want to control the largest component of E {h), the error measure would be 

e{h) = max\E i[h)\ (7.21) 

i 

We could also control some gross measure of the error, such as the root-mean-square 


21 Cash, J. R., and Carp, A. H., ACM Transactions on Mathematical Software, Vol. 16, p. 201 (1990). 
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error defined by 



Y n— 1 


(7.22) 


where nis the number of first-order equations. Then we would use 


e{h) = E {Ii) 


(7.23) 


for the error measure. Since the root-mean-square error is easier to handle, we adopt 
it for our program. 

Error control is achieved by adjusting the increment h so that the per-step error 
e(h ) is approximately equal to a prescribed tolerance e. Noting that the truncation 
error in the fourth-order formula is 0{lf), we conclude that 



(a) 


Let us suppose that we performed an integration step with h\ that resulted in the error 
e{hi). The step size h 2 that we should have used can now be obtained from Eq. (a) by 
setting e{h 2 ) = e: 


If h 2 > h \, we could repeat the integration step with h 2 , but since the error was below 
the tolerance, that would be a waste of a perfectly good resuit. So we accept the 
current step and try h 2 in the next step. On the other hand, if h 2 < h \, we must scrap 
the current step and repeat it with h 2 . As Eq. (b) is only an approximation, it is prudent 
to incorporate a small margin of safety. In our program we use the formula 


1/5 



(7.24) 


Recall that e{h) applies to a single integration step; that is, it is a measure of the local 
truncation error. The all-important global truncation error is due to the accumulation 
of the local errors. What should e be set at in order to achieve a global error tolerance 
£ giobai? Since e{ h) is a conservative estimate of the actual error, setting e = e g i 0 bai will 
usually be adequate. If the number integration steps is large, it is advisable to decrease 
e accordingly. 

Is there any reason to use the nonadaptive methods at ali? Usually no; however, 
there are special cases where adaptive methods break down. For example, adaptive 
methods generally do not work if F(jc, y) contains discontinuities. Because the error 
behaves erratically at the point of discontinuity, the program can get stuck in an infinite 
loop trying to find the appropriate value of h. We would also use a nonadaptive method 
if the output is to have evenly spaced values of x. 
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■ run_kut5 

This module is compatible with run_kut 4 listed in the previous chapter. Any program 
that calls integrate can choose between the adaptive and the nonadaptive methods 
by importing either runikut 5 or run_kut4. The input argument h is the trial value of 
the increment for the first integration step. 

## module run_kut5 

’’’ X,Y = integrate(F,x,y,xStop,h,tol=l.Oe-6). 

Adaptive Runge-Kutta method for solving the 
initial value problem {y>’ = {F(x,{y})}, where 
{ Y > = {y[0],y[1],...y[n-l]>. 
x,y = initial conditions 
xStop = terminal value of x 

h = initial increment of x used in integration 

tol = per-step error tolerance 
F = user-supplied function that returns the 

array F(x,y) = {y’[0],y’[1],...,y’[n-1]}. 

from numarray import array,sum,zeros,Float64 
from math import sqrt 

def integrate(F,x,y,xStop,h,tol=1.0e-6): 

def run_kut5(F,x,y,h): 

# Runge-Kutta-Fehlberg formulas 

C = array([3 7./3 78, 0., 250./621, 125./594, \ 

0., 512./1771]) 

D = arrayC[2825./27648, 0., 18575./48384, \ 

13525./55296, 277./14336, 1./4]) 

n = len(y) 

K = zeros((6,n),type=Float64) 

K[0] = h*F(x,y) 

K[1] = h*F(x + 1./5*h, y + l./5*K[0]) 

K[2] = h*F(x + 3./10*h, y + 3./40*K[0] + 9./40*K[l]) 

K[3] = h*F(x + 3./5*h, y + 3./10*K[0]- 9./10*K[l] \ 

+ 6./5*K[2]) 

K[4] = h*F(x + h, y - ll./54*K[0] + 5./2*K[l] \ 

- 70./27*K[2] + 35./27*K[3]) 

K[5] = h*F(x + 7./8*h, y + 1631./55296*K[0] \ 
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+ 175./512*K[1] + 5 75./13824*K[2] 

+ 44275./110592*K[3] + 253./4096*K[4]) 


\ 


# Initialize arrays {dy> and {E} 

E = zeros((n),type=Float64) 

dy = zeros((n),type=Float64) 

# Compute solution increment {dy} and per-step error {E} 
for i in range(6): 

dy = dy + C[i]*K[i] 

E = E + CC[i] - D[i])*K[i] 

# Compute RMS error e 

e = sqrt(sum(E**2)/n) 
return dy,e 

X = [] 

Y = [] 

X. append(x) 

Y. append(y) 

stopper =0 # Integration stopper(0 = off, 1 = on) 

for i in range(lOOOO): 

dy,e = run_kut5(F,x,y,h) 

# Accept integration step if error e is within tolerance 
if e <= tol: 

y = y + dy 
x = x + h 

X. append(x) 

Y. append(y) 

# Stop if end of integration range is reached 
if stopper == 1: break 

# Compute next step size from Eq. (7.24) 
if e != 0.0 : 

hNext = 0.9*h*(tol/e)**0.2 
else: hNext = h 

# Check if next step is the last one; is so, adjust h 
if (h > 0.0) == C(x + hNext) >= xStop): 

hNext = xStop - x 
stopper = 1 
h = hNext 


return array(X),array(Y) 
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from printSoln import * 
from math import exp 

def F(x,y): 

F = zeros((2),type=Float64) 

F [ 0] = y[l] 

F[ 1] = -9.80665 + 65.351e-3 * y[l]**2 * exp(-10.53e-5*y[0]) 
return F 


x = 0.0 

xStop = 10.0 

y = array([9000, 0.0]) 

h = 0.5 

freq = 1 

X,Y = integrate(F,x,y,xStop,h,1.Oe-2) 
printSoln(X,Y,freq) 

raw_input(’’ \nPress return to exit’’) 


Running the program resulted in the following output: 


x 

0.0000e+000 
5.0000e-001 
2.0 584e+000 
3.4602e+000 
4.8756e+000 
6.5347e+000 
8.6276e+000 
1.0000e+001 


y[ 0 ] 

9.0000e+003 
8.9988e+003 
8.9821e+003 
8.9 581e+003 
8.9312e+003 
8.8989e+003 
8.8 580e+003 
8.8312e+003 


y[ 1 ] 

0.0000e+000 
-4.8043e+000 
-1.5186e+001 
-1.8439e+001 
-1.9322e+001 
-1.9533e+001 
-1.9541e+001 
-1.9519e+001 


The first step was carried out with the prescribed trial value h = 0.5 s. Apparently 
the error was well within the tolerance, so that the step was accepted. Subsequent 
step sizes, determined from Eq. (7.24), were considerably larger. 

Inspecting the output, we see that at t = 10 s the object is moving with the speed 
v = —y = 19.52 m/s at an elevation of y = 8831 m. 


EXAMPLE 7.9 

Integrate the moderately stiff problem 

19 

/ = - T y - 10/ y(0) = -9 yco) = 0 

from x = 0 to 10 with the adaptive Runge-Kutta method and plot the results (this 
problem also appeared in Example 7.7). 
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Solutiori Since we use an adaptive method, diere is no need to worry about the stable 
range of h, as we did in Example 7.7. As long as we specify a reasonable tolerance for 
the per-step error (in this case the default value 10 -6 is fine), the algorithm will find 
the appropriate step size. Here is the program and its output: 

#!/usr/bin/python 
## example7_9 

from numarray import array,zeros,Float64 
from run_kut5 import * 
from printSoln import * 

def F(x,y): 

F = zeros((2),type=Float64) 

F[0] = y[1] 

F[1] = -4.75*y[0] - 10.0*y[l] 
return F 


x = 0.0 

xStop = 10.0 

y = array([-9.0, 0.0]) 

h = 0.1 

freq = 4 

X,Y = integrate(F,x,y,xStop,h) 
printSoln(X,Y,freq) 

raw_input('’ \nPress return to exit’’) 


X 

y[ o ] 

y[ 1 ] 

0.0000e+000 

-9.0000e+000 

0.0000e+000 

9.8941e-002 

-8.8461e+000 

2.66 51e+000 

2.1932e-001 

-8.4511e+000 

3.6653e+000 

3.7058e-001 

-7.8784e+000 

3.8061e+000 

5.7229e-001 

-7.1338e+000 

3.5473e+000 

8.6922e-001 

-6.1513e+000 

3.0745e+000 

1.4009e+000 

-4.7153e+000 

2.3 5 77e+000 

2.8 5 58e+000 

-2.2783e+000 

1.1391e+000 

4.3990e+000 

-1.0531e+000 

5.26 56e-001 

5.9545e+000 

-4.8385e-001 

2.4193e-001 

7.5596e+000 

-2.168 5e-001 

1.0843e-001 

9.1159e+000 

-9.9591e-002 

4.9794e-002 

1.0000e+001 

-6.4010e-002 

3.2005e-002 
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EXAMPLE 7.8 

The aerodynamic drag force acdng on a certain object in free fall can be approximated 
by 

F d = ai/e~ by 

where 

v = velocity of the object in m/s 
y = elevation of the object in meters 
a = 7.45 kg/m 
b= 10.53 x 10“ 5 m -1 

The exponential term accounts for the change of air density with elevation. The dif- 
ferential equation describing the fall is 

my= —mg+ F D 

where g= 9.80665 m/s 2 and m= 114 kg is the mass of the object. If the object is 
released at an elevation of 9 km, determine its elevation and speed after a 10-s fall 
with the adaptive Runge-Kutta method. 

Solution The differential equation and the initial conditions are 
y= ~g+ — T 2 exp(-by) 

7.45 

= —9.80665+ -y^j-y 2 exp(—10.53 x 10 _5 y) 
y(0) = 9000 m y(0) = 0 

Letting y 0 = y and yi = y, we obtain the equivalent first-order equations as 


yo 


yi 

_h_ 


-9.80665 + (65.351 x 10“ 3 ) y 2 exp(-10.53 x lQ- 5 y 0 ) 


y(0) 


9000 m 
0 


The driver program for run.kut 5 is listed below. We specihed a per-step error toler- 
ance of 10 -2 in integrate. Considering the magnitude of y, this should be enough 
for five decimal point accuracy in the solution. 


#!/usr/bin/python 
## example7_8 

from numarray import array,zeros,Float64 
from run_kut5 import * 
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The results are in agreement with the analytical solution. 

The plots of y and / show every fourth integration step. Note the high density of 
points near x = 0 where / changes rapidly. As the /-curve becomes smoother, the 
distance between the points increases. 



7.6 Bulirsch-Stoer Method 
Midpoint Method 

The midpoint formula of numerical integration of / = F(x, y) is 

y(x + h) = y(x — h) + 2h¥ [x, y(x)] (7.25) 

It is a second-order formula, like the modified Euler’s formula. We discuss it here 
because it is the basis of the powerful Bulirsch-Stoer method, which is the technique 
of choice in problems where high accuracy is required. 



Figure 7.3. Graphical repesentation of the midpoint 
formula. 


Figure 7.3 illustrates the midpoint formula for a single differential equation 
/ = /(x, y). The change in y over the two panels shown is 

/>x+h 

y(x + h) — y(x — h) = I y{x)dx 
J x—h 


which equals the area under the / (x) curve. The midpoint method approximates this 
area by the area 2/z/(x, y) of the cross-hatched rectangle. 
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x 0 x, x 2 x 3 v x n .i x n 


Figure 7.4. Mesh used in die midpoint method. 


Consider now advancing the solution of y (x) = F (x, y) from x = xo to xo + H with 
the midpoint formula. We divide the interval of integration into n steps of length 
h= H/n each, as shown in Fig. 7.4, and carry out the computations 

yi = yo + /2F 0 
y2 = yo + 2/2F1 

y3 = yi + 2 / 2 F 2 (7.26) 


Yn = Yn-2 + 2hF„_i 

Here we used the notation y,- = y(x,-) and F; = F(x,-, y,). The First of Eqs. (7.26) uses 
the Euler formula to “seed” the midpoint method; the other equations are midpoint 
formulas. The final resuit is obtained by averaging y„ in Eq. (7.26) and the estimate 
y n ~ y„_i + / 2 F„ available from Euler formula: 

y(xb + H) = * [y„ + (y„_i + h¥ n j] (7.27) 


Richardson Extrapolation 

It can be shown that the error in Eq. (7.27) is 

E = Cih 2 + C 2 I 2 4 + c 3 h 6 H- 


Herein lies the great utility of the midpoint method: we can eliminate as many of the 
leading error terms as we wish by Richardson’s extrapolation. For example, we could 
compute y(xb + H) with a certain value of h and then repeat the process with h/2. 
Denoting the corresponding results by g(/ 2 ) and Richardson’s extrapolation— 

see Eq. (5.9)—then yields the improved resuit 


ybetter (-^0 + H) 


4g(h/2) - g(/2) 
3 


which is fourth-order accurate. Another round of integration with h /4 followed by 
Richardson’s extrapolation get us sixth-order accuracy, etc. 

The y’s in Eqs. (7.26) should be viewed as a intermediate variables, because unlike 
y(xb + H), they cannot be refined by Richardson’s extrapolation. 
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■ midpoint 

The function midpoint in this module combines the midpoint method with 
Richardson extrapolation. The first application of the midpoint method uses two 
integration steps. The number of steps is doubled in successive integrations, each 
integration being followed by Richardson extrapolation. The procedure is stopped 
when two successive Solutions differ (in the root-mean-square sense) by less than a 
prescribed tolerance. 

## module midpoint 

’’’ yStop = integrate (E,x,y,xStop,tol=l.Oe-6) 

Modified midpoint method for solving the 

initial value problem y’ = F(x,y>. 

x,y = initial conditions 

xStop = terminal value of x 

yStop = y(xStop) 

F = user-supplied function that returns the 

array F(x,y) = {y’[0],y’[1],...,y’[n-l]>. 

from numarray import zeros,Float64,sum 
from math import sqrt 

def integrate(F,x,y,xStop,tol): 

def midpoint(F,x,y,xStop,nSteps): 

# Midpoint formulas 

h = (xStop - x)/nSteps 
yo = y 

yl = yO + h*F(x,yO) 
for i in range(nSteps-l): 
x = x + h 

y2 = yO + 2.0*h*F(x,yl) 
yO = yl 
Yl = y2 

return 0.5*(yl + yO + h*F(x,y2)) 

def richardson(r,k): 

# Richardson's extrapolation 

for j in range(k-l,0,-1): 
const = 4.0**(k-j) 
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r[j] = (const*r[j+1] - r[j])/(const - 1.0) 
return 

kMax = 51 
n = len(y) 

r = zeros((kMax,n),type=Float64) 

# Start with two integration steps 
nSteps = 2 

r[l] = midpoint(F,x,y,xStop,nSteps) 
r_old = r[l].copy() 

# Double the number of integration points 

# and refine resuit by Richardson extrapolation 
for k in range(2,kMax): 

nSteps = nSteps*2 

r[k] = midpoint(F,x,y,xStop,nSteps) 
richardson(r,k) 

# Compute RMS change in solution 

e = sqrt(sum((r[1] - r_old)**2)/n) 

# Check for convergence 
if e < tol: return r[l] 
r_old = r[l].copy() 

print ''Midpoint method did not converge’’ 


Bulirsch-Stoer Algorithm 

When used on its own, the module midpoint has a major shortcoming: the solution 
at points between the initial and final values of x cannot be refined by Richardson 
extrapolation, so that y is usable only at the last point. This deficiency is rectified in 
the Bulirsch-Stoer method. The fundamental idea behind the method is simple: apply 
the midpoint method in a piecewise fashion. That is, advance the solution in stages of 
length H, using the midpoint method with Richardson extrapolation to perform the 
integration in each stage. The value of H can be quite large, since the precision of the 
resuit is determined by the step length h in the midpoint method, not by H. 

The original Bulirsch and Stoer technique 22 is a complex procedure that incorpo- 
rates many refinements missing in our algorithm. However, the function bulStoer 
given below retains the essential ideas of Bulirsch and Stoer. 

What are the relative merits of adaptive Runge-Kutta and Bulirsch-Stoer meth- 
ods? The Runge-Kutta method is more robust, having higher tolerance for nonsmooth 


22 Stoer, J., and Bulirsch, R., Introduction to Numerical Analysis, Springer, 1980. 
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functions and stiff problems. In most applications where high precision is not required, 
it also tends to be more efficient. However, this is not the case in the computation of 
high-accuracy Solutions involving smooth functions, where the Bulirsch-Stoer algo- 
rithm shines. 

■ bulStoer 

This function contains a simplified algorithm for the Bulirsch-Stoer method. 

## module bulStoer 

’’’ X,Y = bulStoer(F,x,y,xStop,H,tol=l.Oe-6) . 

Simplified Bulirsch-Stoer method for solving the 
initial value problem {y>’ = {F(x,{y>)>, where 

fy> = {y[0],y[l]-y[n-l]>. 

x,y = initial conditions 
xStop = terminal value of x 

H = increment of x at which results are stored 

F = user-supplied function that returns the 

array F(x,y) = {y’[0] ,y’ [1], ... ,y’[n-l]> . 

from numarray import array 
from midpoint import * 

def bulStoer(F,x,y,xStop,H,tol=1.0e-6): 

X = [] 

Y = [] 

X. append(x) 

Y. append(y) 
while x < xStop: 

H = min(H,xStop - x) 

y = integrate(F,x,y,x + H,tol) # Midpoint method 
x = x + H 

X. append(x) 

Y. append(y) 

return array(X),array(Y) 


EXAMPLE 7.10 

Compute the solution of the initial value problem 


y = sin y y( 0) = 1 
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at x = 0.5 with the midpoint formulas using n= 2 and n = 4, followed by Richardson 
extrapolation (this problem was solved with the second-order Runge-Kutta method 
in Example 7.3). 

Solution With n= 2 the step length is h= 0.25. The midpoint formulas, Eqs. (7.26) 
and (7.27), yield 

34 = y 0 + hfo = 1 + 0.25 sin 1.0 = 1.210368 
>4 = y n + 2hf = 1 + 2(0.25) sin 1.210368= 1.467 87 3 

3^(0. 5 ) = ^(34 + y 0 + hf 2 ) 

= ^ (1.210 368 + 1.467 87 3 + 0.25 sin 1.467 87 3) 

= 1.463 459 

Using n = 4 we have h = 0.125 and the midpoint formulas become 
34 = yo + hf 0 = 1 + 0.125 sin 1.0 = 1.105184 
yz = yo + 2/i/i = 1 + 2(0.125) sin 1.105 184= 1.223387 
34 = yi+2hf 2 = 1.105184 + 2(0.125) sin 1.223 387 = 1.340248 
y 4 = 34 + 2 / 1/3 = 1.223 387 + 2(0.125) sin 1.340 248= 1.466 772 

34,72(0.5) = ^( 3/4 + y 3 + hf A ) 

1 

= - (1.466 772 + 1.340 248 + 0.125 sin 1.466 772) 

= 1.465 672 

Richardson extrapolation results in 

= 4jv 2 (0.5)-J4,(0.5) = 4(1.465672)- 1.463459 = 

7 3 3 

which compares favorably with the “true” solution y(0.5) = 1.466404. 

EXAMPLE 7.11 


L 
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The differential equations governing the loop current i and the charge q on the ca- 
pacitor of the electric Circuit shown are 


di q 

L— + Ri+l = E[fi 
dt C 


dq 

dt 


If the applied voltage E is suddenly increased from zero to 9 V, plot the resulting loop 
current during the first ten seconds. Use R= 1.0 £2, L = 2H and C = 0.45F. 


Solution Letting 

q 

i 

and substituting the given data, the differential equations become 



yo 


yi 



{-Ryi - y 0 /C+ E)/L 


The initial conditions are 


y(0) = 


0 

0 


We solved the problem with the function buistoer with the increment 
H = 0.5 s: 


## example7_ll 

from bulStoer import * 

from numarray import array,zeros,Float64 
from printSoln import * 

def F(x,y): 

F = zeros((2),type=Float64) 

F [ 0] = y[l] 

F[1] = (-y[1] - y[0]/0.45 + 9.0)/2.0 
return F 


H = 0.5 
xStop = 10.0 
x = 0.0 

y = array([0.0, 0.0]) 

X,Y = bulStoer(F,x,y,xStop,H) 
printSoln(X,Y,1) 

raw_input(’’ \nPress return to exit’’) 
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Skipping the numerical output, the plot of the current is 



Recall that in each interval H (the spacing of open circles) the integration was per- 
formed by the modified midpoint method and refined by Richardson’s extrapolation. 

PROBLEM SET 7.2 

1. Derive the analytical solution of the problem 

/ + / - 380y = 0 y(0) = i /(0) = -20 

Would you expect difficulties in solving this problem numerically? 

2. Consider the problem 


y = x — 10y y(0) = 10 

(a) Verify that the analytical solution is y{x) = 0.1 jc - 0.01 + 10.0le -10 *. 

(b) Determine the step size h that you would use in numerical solution with 
the (nonadaptive) Runge-Kutta method. 

3. ■ Integrate the initial value problem in Prob. 2 from x = 0 to 5 with the Runge- 
Kutta method using (a) h = 0.1, (b) h= 0.25 and (c) h= 0.5. Comment on the 
results. 

4. ■ Integrate the initial value problem in Prob. 2 from x = 0 to 10 with the adaptive 
Runge-Kutta method. 



5. ■ 
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The differential equation describing the motion of the mass-spring-dashpot Sys¬ 
tem is 


y+ 


c . k 

—y+ — y= ° 

m m 


wher em= 2kg ,c= 460N-s/mandA:= 450N/m. The initial conditions are y(0) = 
0.01 m and y(0) = 0. (a) Show that this is a stiff problem and determine a value of 
h that you would use in numerical integration with the nonadaptive Runge-Kutta 
method. (b) Carry out the integration from t = 0 to 0.2 s with the chosen h and 
plot y vs. t. 

6 . ■ Integrate the initial value problem specified in Prob. 5 with the adaptive Runge- 
Kutta method from t = 0 to 0.2 s and plot y vs. t. 

7. ■ Compute the numerical solution of the differential equation 


/ = 16.81y 


from x = 0 to 2 with the adaptive Runge-Kutta method. Use the initial conditions 
(a) y(0) = 1.0, /(0) = —4.1; and (b) y(0) = 1.0, /(0) = —4.11. Explain the large 
difference in the two Solutions. Hint: derive the analytical Solutions. 

8 . ■ Integrate 

/ + /-/ = 0 y(0) = 1 yco) = 0 


from x = 0 to 3.5. Is the sudden increase in y near the upper limit is real or an 
artifact caused by instability? 

9. ■ Solve the stiff problem—see Eq. (7.16) 


y" + 1001/ + lOOOy = 0 y(0) = 1 /(0) = 0 


from x = 0 to 0.2 with the adaptive Runge-Kutta method and plot / vs. x. 

10. ■ Solve 


/ + 2/ + 3y = 0 y(0) = 0 /(0) = V2 

with the adaptive Runge-Kutta method from x = 0 to 5 (the analytical solution is 
y= e _j: sin/2x). 

11. ■ Solve the differential equation 


/ = 2 y/ 

from x = 0 to 10 with the initial conditions y(0) = 1, /(0) = — 1. Plot y vs. x. 

12. ■ Repeat Prob. 11 with the initial conditions y(0) = 0, /(0) = 1 and the integration 
range x = 0 to 1.5. 
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13. ■ Use the adaptive Runge-Kutta method to integrate 

y = (- y -y) x y(°) = 5 

from x = 0 to 4 and plot y vs. x. 

14. ■ Solve Prob. 13 with the Bulirsch-Stoer method using H = 0.5. 

15. ■ Integrate 

x 2 / + xy + y=0 y(l) = 0 /(1) =-2 

from x = 1 to 20, and plot y and / vs. x. Use the Bulirsch-Stoer method. 

16. ■ 



The magnetized iron block of mass m is attached to a spring of stiffness k and 
free length L. The block is at rest at x = L when the electromagnet is turned on, 
exerting the repulsive force F = c/x 2 on the block. The differential equation of 
the resulting motion is 

c 

mx = r — k{x — L ) 

Determine the amplitude and the period of the motion by numerical integration 
with the adaptive Runge-Kutta method. Use c = 5 Nm 2 , k= 120 N/m, L = 0.2 m 
and m= 1.0 kg. 

17. ■ 


4 

The bar ABC is attached to the vertical rod with a horizontal pin. The assembly 
is free to rotate about the axis of the rod. In the absence of friction, the equations 
of motion of the system are 

0 = cj ) 2 sine cos 6 0 = —26*0 cotff 

The system is set into motion with the initial conditions 6* (0) = .t /12 rad, 
@(0) = 0, 0(0) = 0 and 0(0) = 20 rad/s. Obtain a numerical solution with the 
adaptive Runge-Kutta method from t = 0 Lo 1.5 s and plot 0 vs. L 
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18. ■ Solve the Circuit problem in Example 7.11 if R = 0 and 

£ M ,| 0when ' <0 

I 9 sin7rtwhen t > 0 

19. ■ Solve Prob. 21 in Problem Set 1 if E = 240 V (constant). 

20. ■ 



Kirchoffs equations for the Circuit in the figure are 

dii 


L—— + R\i\ + R 2 [i\ — h) = E(t) 
dt 

dio cto 

L-£ + R 2 {i 2 - h) + ^ = 0 
dt C 


where 


dq 2 
dt 


= i 2 


Using the data R\ = 4 Q, R 2 = 10 £2, L = 0.032 H, C = 0.53 F and 

. 20 VifO < t< 0.005 s 
E{t) = { 

0 otherwise 


plot the transient loop currents i\ and i 2 from t = 0 to 0.05 s. 

21. ■ Consider a closed biological system populated by M number of prey and N 
number of predators. Volterra postulated that the two populations are related by 
the differential equations 


M= aM- bMN 
N= -cN+dMN 


where a, b, c and d are constants. The steady-state solution is Mq = c/d, N 0 = a/b, 
if numbers other than these are introduced into the system, the populations 
undergo periodic fluctuations. Introducing the notation 


y 0 = M/Mq yi = N/N 0 
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allows us to write the differential equations as 

3>o = a{y Q - y 0 yi) 
yi = b{-y 1 + y 0 yi) 

Using a = 1.0/year, b= 0.2/year, y 0 (0) = 0.1 and yi(0) = 1.0, plot the two popu- 
lations from t = 0 to 50 years. 

22. ■ The equations 


u = —au+ av 


v = cu— v— uw 


w = —bw+ uv 

known as the Lorenz equations, are encountered in theory of fluid dynamics. 
Letting a = 5.0, b = 0.9 and c = 8.2, solve these equations from t = 0 to 10 with 
the initial conditions u{ 0) = 0, v(0) = 1.0, w(0) = 2.0 and plot u{t). Repeat the 
solution with c = 8.3. What conclusions can you draw from the results? 


7.7 Other Methods 

The methods described so far belong to a group known as single-step methods. The 
name stems from the fact that the information at a single point on the solution curve 
is sufficient to compute the next point. There are also multistep methods that utilize 
several points on the curve to extrapolate the solution at the next step. Well-known 
members of this group are the methods of Adams, Milne, Hamming and Gere. These 
methods were popular once, but have lost some of their luster in the last few years. 
Multistep methods have two shortcomings that complicate their implementation: 

• The methods are not self-starting, but must be provided with the solution at the 
first few points by a single-step method. 

• The integration formulas assume equally spaced steps, which makes it makes it 
difficult to change the step size. 

Both of these hurdles can be overcome, but the price is complexi ty of the algorithm 
that increases with the sophistication of the method. The benefits of multistep meth¬ 
ods are minimal—the best of them can outperform their single-step counterparts in 
certain problems, but these occasions are rare. 
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Solve /' = f{x, y, /), y(a) = a, y(b) = /3 


8.1 Introduction 

In two-point boundary value problems the auxiliary conditions associated with the 
differential equation, called the boundary conditions, are specified at two different 
values of x. This seemingly small departure from initial value problems has a major 
repercussion—it makes boundary value problems considerably more difficult to solve. 
In an initial value problem we were able to start at the point where the initial values 
were given and march the solution forward as far as needed. This technique does not 
work for boundary value problems, because there are not enough starting conditions 
available at either end point to produce a unique solution. 

One way to overcome the lack of starting conditions is to guess the missing values. 
The resuiting solution is very unlikely to satisfy boundary conditions at the other end, 
but by inspecting the discrepancy we can estimate what changes to make to the initial 
conditions before integrating again. This iterative procedure is known as the shooting 
method. The name is derived from analogy with target shooting—take a shot and 
observe where it hits the target, then correct the aim and shoot again. 

Another means of solving two-point boundary value problems is th e finite differ- 
ence method, where the differential equations are approximated by finite differences 
at evenly spaced mesh points. As a consequence, a differential equation is transformed 
into set of simultaneous algebraic equations. 

The two methods have a common problem: they give rise to nonlinear sets of 
equations if the differential equations are not linear. As we noted in Chapter 2, all 
methods of solving nonlinear equations are iterative procedures that can consume a 
lot of computational resources. Thus solution of nonlinear boundary value problems 
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is not cheap. Another complication is that iterative methods need reasonably good 
starting values in order to converge. Since there is no set formula for determining these, 
an algorithm for solving nonlinear boundary value problems requires informed input; 
it cannot be treated as a “black box.” 


8.2 Shooting Method 

Second-Order Differential Equation 

The simplest two-point boundary value problem is a second-order differential equa¬ 
tion with one condition specified at x = a and another one at x = b. Here is an example 
of such a problem: 

/ = f(x, y, y), y(a) = a, y(b) = p (8.1) 

Let us now attempt to turn Eqs. (8.1) into the initial value problem 

/ = f(x, y, /), y(a) = a, /( a ) = u (8.2) 

The key to success is finding the correct value of u. This could be done by trial and 

error: guess u and solve the initial value problem by marching from x = a to b. If 

the solution agrees with the prescribed boundary condition y{b ) = fi, we are done; 
otherwise we have to adjust u and try again. Clearly, this procedure is very tedious. 

More systematic methods become available to us if we realize that the determi- 
nation of uis a root-finding problem. Because the solution of the initial value problem 
depends on u, the computed value of y(b) is a function of u; that is 

y(b) = 0{u ) 

Hence u is a root of 


r(u)=6(u)~P = 0 (8.3) 

where r[ii) is the boundary residual (difference between the computed and specified 
boundary value at x= b). Equation (8.3) can be solved by one of the root-finding 
methods discussed in Chapter 4. We reject the method of bisection because it involves 
too many evaluations of 0 ( u). In the Newton-Raphson method we run into the problem 
of having to compute dB/du, which can be done, but not easily. That leaves Brent’s 
algorithm as our method of choice. 

Here is the procedure we use in solving nonlinear boundary value problems: 

1. Specify the starting values U\ and u 2 which must bracket the rootu of Eq. (8.3). 

2. Apply Brent’s method to solve Eq. (8.3) for u. Note that each iteration requires 
evaluation of 6 (u) by solving the differential equation as an initial value problem. 
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3. Having determined the value of u, solve the differential equations once more and 
record the results. 

If the differential equation is linear, any root-finding method will need only one 
interpolation to determine u. But since Brent’s method uses quadratic interpolation, 
it needs three points: U\, u 2 and w, , the latter being provided by a bisection step. This 
is wasteful, since linear interpolation with U\ and u 2 would also resuit in the correct 
value of u. Therefore, we replace Brent’s method with linear interpolation whenever 
the differential equation is linear. 

■ linlnterp 

Here is the algorithm we use for linear interpolation: 

## module linlnterp 

’’’ root = linlnterp(f,xl,x2). 

Finds the zero of the linear function f(x) by straight 
line interpolation based on x = xl and x2. 

def linlnterpff,xl,x2): 
fl = f(xl) 
f2 = f(x2) 

return = x2 - f2*(x2 - xl)/(f2 - fl) 

EXAMPLE 8.1 

Solve the boundary value problem 

/ + 3y/ = 0 y(0) = 0 y( 2) = 1 

Solution The equivalent first-order equations are 



yi 

3y 0 yi 


with the boundary conditions 


yo(0)=0 y 0 (2) = 1 


Now comes the daunting task of determining the trial values of /(0). We could 
always pick two numbers at random and hope for the best. However, it is possible 
to reduce the element of chance with a little detective work. We start by making the 
reasonable assumption that y is smooth (does not wiggle) in the interval 0 < x < 2. 
Next we note that y has to increase from 0 to 1, which requires j/ > 0. Since both y and 









298 


Two-Point Boundary Value Problems 


y are positive, we conclude that y" must be negative in order to satisfy the differential 
equation. Now we are in a position to make a rough sketch of y: 



Looking at the sketch it is ciear that /(0) > 0.5, so that /(0) = 1 and 2 appear to be 
reasonable values for the brackets of /(0); if they are not, Brent’s method will display 
an error message. 

In the program listed below we chose the fourth-order Runge-Kutta method 
for integration. It can be replaced by the adaptive version by substituting run_kut5 
for run_kut4 in the import statement. Note that three user-supplied functions are 
needed to describe the problem at hand. Apart from the function F(x,y) that de- 
fines the differential equations, we also need the functions initCond(u) to specify 
the initial conditions for integration, and r(u) to provide Brent’s method with the 
boundary condition residual. By changing a few statements in these functions, the 
program can be applied to any second-order boundary value problem. It also works 
for third-order equations if integration is started at the end where two of the three 
boundary conditions are specified. 

#!/usr/bin/python 
## example8_l 

from numarray import zeros,Float64,array 
from run_kut4 import * 
from brent import * 
from printSoln import * 

def initCond(u): # Init, values of [y, y’]; use ’u’ if unknown 

return array([0.0, u]) 

def r(u): # Boundary condition residual--see Eq. (8.3) 

X,Y = integrate(F,xStart,initCond(u),xStop,h) 
y = Y[len(Y) - 1] 
r = y[0] - 1.0 
return r 

def F(x,y): # First-order differential equations 

F = zeros((2),type=Float64) 

F[0] = y[1] 
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F[l] = -3.0*y[0]*y[l] 
return F 


xStart = 0.0 
xStop = 2.0 
ul = 1.0 
u2 = 2.0 
h = 0.1 
freq = 2 


# Start of integration 

# End of integration 

# lst trial value of unknown init. cond. 

# 2nd trial value of unknown init. cond. 

# Step size 

# Printout frequency 


u = brent(r,ul,u2) # Compute the correct initial condition 

X,Y = integrate(F,xStart,initCond(u),xStop,h) 
printSoln(X,Y,freq) 

raw_input(’’ \nPress return to exit’’) 


Here is the solution : 


x 

0.0000e+000 
2.0000e-001 
4.0000e-001 
6.0000e-001 
8.0000e-001 
1.0000e+000 
1.2000e+000 
1.4000e+000 
1.6000e+000 
1.8000e+000 
2.0000e+000 


y[ 0 ] 

0.0000e+000 
2.9404e-001 
5.4170e-001 
7.218 7e-001 
8.3944e-001 
9.1082e-001 
9.522 7e-001 
9.7572e-001 
9.8880e-001 
9.9602e-001 
1.0000e+000 


y[ 1 ] 

1.5145e+000 
1.3848e+000 
1.0743e+000 
7.3287e-001 
4.5752e-001 
2.7013e-001 
1.5429e-001 
8.6471e-002 
4.7948e-002 
2.6430e-002 
1.4522e-002 


Note that /(0) = 1.5145, so that our starting values of 1.0 and 2.0 were on the 
mark. 

EXAMPLE 8.2 

Numerical integration of the initial value problem 

/ + 4y = 4x y(0) = 0 j/(0) = 0 

yielded y (2) = 1.653 64. Use this information to determine the value of / (0) that would 
resuit in /(2) = 0. 

Solution We use linear interpolation 

u 2 - Ui 


u= u 2 - 9 {u 2 ) 


9{u 2 ) -9{u i) 
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where in our case u = /(0) and 0[u) = /(2). So far we are given ii\ = 0 and 6{U\) = 
1.653 64. To obtain the second point, we need another solution of the initial value 
problem. An obvious solution is y = x, which gives us y(0) = 0 and /(0) = /(2) = 1. 
Thus the second point is u 2 = 1 and B(ii 2 ) = 1. Linear interpolation nowyields 

/(0) = u= 1 - (1) - 1 ~° = 2.529 89 

1 — 1.653 64 


EXAMPLE 8.3 

Solve the third-order boundary value problem 

/" = 2/ + 6xy y(0) = 2 y(5) = /(5) = 0 

and plot y vs. x. 

Solution The first-order equations and the boundary conditions are 


/o 


yi 

/l 

= 

T2 

./ 2 . 


_2y 2 + 6xy 0 _ 


y 0 (0)=2 yo (5) = yi (5) = 0 

The program listed below is based on example8_l. Because two of the three 
boundary conditions are specified at the right end, we start the integration at x = 5 
and proceed with negative h Loward x = 0. Two of the three initial conditions are pre- 
scribed: y 0 (5) = yi(5) = 0, whereas the third condition y 2 (5) is unknown. Because the 
differential equation is linear, we replaced brent with linlnterp. In linear interpo¬ 
lation the two guesses for y 2 (5) { u\ and u 2 ) are not important, so we left them as they 
were in Example 8.1. The adaptive Runge-Kutta method (run_kut5) was chosen for 
the integration. 

#!/usr/bin/python 
## example8_3 

from numarray import zeros,Float64,array 
from run_kut5 import * 
from linlnterp import * 
from printSoln import * 

def initCond(u): # Initial values of [y,y’,y’’l; 

# use ’u’ if unknown 
return array([0.0, 0.0, u]) 

def r(u): # Boundary condition residual--see Eq. (8.3) 

X,Y = integrate(F,xStart,initCond(u),xStop,h) 
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y = Y[len(Y) - 1] 
r = y[0] - 2.0 
return r 


def F(x,y): # Eirst-order differential equations 

F = zeros((3),type=Float64) 

F[0] = y[l] 

F[1] = y[2] 

F[2] = 2.0*y[2] + 6.0*x*y[0] 
return F 


xStart = 5.0 
xStop = 0.0 
ul = 1.0 
u2 = 2.0 
h = -0.1 
freq = 2 


# Start of integration 

# End of integration 

# lst trial value of unknown init. cond. 

# 2nd trial value of unknown init. cond. 

# initial step size 

# printout frequency 


u = linlnterpCr,ul,u2) 

X,Y = integrate(F,xStart,initCond(u),xStop,h) 
printSoln(X,Y,freq) 

raw_input('’ \nPress return to exit’’) 


We forgo the rather long printout of the solution and show just the plot: 



Higher-Order Equations 

Let us consider the fourth-order differential equation 

y 4) = /(x,y, 


(8.4a) 
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with the boundary conditions 


yia) = a i y"(a) = a 2 y(b) = p 1 f{b) = p 2 (8.4b) 

To solve Eq. (8.4a) with the shooting method, we need four initial conditions at x = a, 
only two ofwhich are specified. Denotingthe unknown initial values by U\ and u 2 , we 
have the set of initial conditions 


y{a) = »1 /(«) = U[ y"(a ) = a 2 /"(a) = u 2 (8.5) 

If Eq. (8.4a) is solved with the shooting method using the initial conditions in Eq. (8.5), 
the computed boundary values at x = fodepend on the choice of U\ and u 2 . We denote 
this dependence as 


y{b) = 0i(Mi, u 2 ) f{b) = 0 2 {ui, u 2 ) (8.6) 

The correct values U\ and u 2 satisfy the given boundary conditions at x = b: 

0\{ui,u 2 )=p 1 0 2 (u\, u 2 ) = /S 2 


or, using vector notation 


0(u) = /3 (8.7) 

These are simultaneous (generally nonlinear) equations that can be solved by the 
Newton-Raphson method discussed in Section 4.6. It must be pointed out again that 
intelligent estimates of U\ and u 2 are needed if the differential equation is not linear. 

EXAMPLE 8.4 



The displacement v of the simply supported beam can be obtained by solving the 
boundary value problem 


(ftv Wq x 
~dx 4 “ UT 


v = 


cfv 

dx 2 


= 0 at x = 0 and x = 


L 


where EI is the bending rigidity. Determine by numerical integration the slopes at 
the two ends and the displacement at mid-span. 


Solution Introducing the dimensionless variables 


1 = 


x 

L 


y= 


EI 

WqL a V 







303 


8.2 Shooting Method 


the problem is transformed to 




= H 



0 at f = 0 and 1 


The equivalent first-order equations and the boundary conditions are (the prime 
denotes d/d%) 


~/o’ 


~yi~ 

/l 


T2 

/2 


T 3 

-/3- 


J _ 


yo(0) = y 2 (0) = j/ 0 (D = y 2 (D = o 


The program listed below is similar to the one in Example 8.1. With appropri- 
ate changes in functions F(x,y), initCond(u) and r(u) the program can solve 
boundary value problems of any order greater than two. For the problem at hand 
we chose the Bulirsch-Stoer algorithm to do the integration because it gives us con- 
trol over the printout (we need y precisely at mid-span). The nonadaptive Runge- 
Kutta method could also be used here, but we would have to guess a suitable step 
size h. 

As the differential equation is linear, the solution requires only one iteration with 
the Newton-Raphson method. In this case the initial values U\ = dy/d%\ x=Q and u 2 = 
cPy/d^ 3 \ x=0 are irrelevant; convergence always occurs in one iteration. 


#!/usr/bin/python 
## example8_4 

from numarray import zeros,Float64,array 
from bulStoer import * 
from newtonRaphson2 import * 
from printSoln import * 

def initCond(u): # Initial values of [y,y’ ,y’’,y’’ ’ ] ; 

# use ’u’ if unknown 
return array([0.0, u[0], 0.0, u[l]]) 

def r(u): # Boundary condition residuals--see Eq. (8.7) 

r = zeros(len(u),type=Float64) 

X,Y = bulStoer(F,xStart,initCond(u),xStop,H) 
y = Y[len(Y) - 1] 
r[0] = y[0] 
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r [ 1] = y [2] 
return r 

def F(x,y): # First-order differential equations 

F = zeros((4),type=Float64) 

F[0] = y[1] 

F [ 1] = y [2] 

F[2] = y[3] 

F [ 3 ] = x 
return F 

xStart =0.0 # Start of integration 

xStop =1.0 # End of integration 

u = array([0.0, 1.0]) # Initial guess for {u> 

H = 0.5 # Printout increment 

freq =1 # Printout frequency 

u = newtonRaphson2(r,u,1.0e-4) 

X,Y = bulStoer(F,xStart,initCond(u),xStop,H) 
printSoln(X,Y,freq) 

raw_input('’ \nPress return to exit’’) 

Here is the output: 

x y[ o ] y[ i ] y[ 2 ] y[ 3 ] 

0.OOOOe+OOO 0.0000e+000 1.9444e-002 0.0000e+000 -1.6667e-001 

5.0000e-001 6.5104e-003 1.2153e-003 -6.2500e-002 -4.1667e-002 

1.OOOOe+OOO -2.4670e-014 -2.2222e-002 -2.7190e-012 3.3333e-001 

Noting that 



we obtain 


dv 


= 19.444 x 10“ 3 


u^L 3 


EI 


dv 

dx x=L 


= -22.222 x 10“ 3 


.3 U>qL 3 


EI 
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which agree with the analytical solution (easily obtained by direct integration of the 
differential equation). 

EXAMPLE 8.5 

Solve 

y {4) + -/ = o 

X 

with the boundary conditions 

y(0) = yco) = o / (i) = o /'(1) = i 

and plot y vs. x. 

Solution Our first task is to handle the indeterminacy of the differential equation 
at the origin, where x = y = 0. The problem is resolved by applying L’Hospital’s rule: 
4y°’/x -*■ 12y 2 / as x —»• 0. Thus the equivalent first-order equations and the boundary 
conditions that we use in the solution are 




yi 

~/o~ 


k2 

k3 

| — 12>gyi if x = 0 

y 


y 

-/ 3 - 



I —4j^/x otherwise 


yo(0) = yi(0)=0 y 2 (l) = 0 y(l) = l 

Because the problem is nonlinear, we need reasonable estimates for /'(0) and 
/"(O).On the basis of the boundary conditions /'(1) =0and/"(l) = 1, the plot ofy" 
is likely to look something like this: 



Ifwe are right, then /'(0) < 0 and /"(0) > 0. Based on this rather scanty information, 
we try y"(0) = -1 and /"(0) = 1. 

The following program uses the adaptive Runge-Kutta method (run_kut5) for 
integration: 

#!/usr/bin/python 
## example8_5 

from numarray import zeros,Float64,array 










306 


Two-Point Boundary Value Problems 


from run_kut5 import * 
from newtonRaphson2 import * 
from printSoln import * 


def initCond(u): # Initial values of [y,y’,y’’,y’’’]; 

# use ’u’ if unknown 
return array([0.0, 0.0, u[0], u[l]]) 

def r(u): # Boundary condition residuals-- see Eq. (8.7) 

r = zeros(len(u),type=Float64) 

X,Y = integrate(F,x,initCond(u),xStop,h) 

y = Y[len(Y) - 1] 

r [ 0 ] = y [ 2 ] 

r[l] = y[3] - 1.0 

return r 


def F(x,y): # First-order differential equations 

F = zeros((4),type=Float64) 

F[0] = y[1] 

F[l] = y[2] 

F[2] = y[3] 

if x < 10.e-4: F[3] = -12.0*y[l]*y[0]**2 
else: F[3] = -4.0*(y[0]**3)/x 

return F 


x = 0.0 
xStop = 1.0 

u = array([-1.0, 1.0]) 
h = 0.1 
freq = 1 


# Start of integration 

# End of integration 

# Initial guess for u 

# Initial step size 

# Printout frequency 


u = newtonRaphson2(r,u,1.Oe-5) 

X,Y = integrate(F,x,initCond(u),xStop,h) 
printSoln(X,Y,freq) 

raw_input('’ \nPress return to exit’’) 


The results are: 


x 

0.0000e+000 
1.0000e-001 


y[ 0 ] 

0.0000e+000 
-4.7184e-003 


y[ i ] 

0.OOOOe+OOO 
-9.2750e-002 


y[ 2 ] 

-9.7607e-001 
-8.7893e-001 


y[ 3 ] 

9.7131e-001 
9.7131e-001 
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3.9576e-001 
7.0683e-001 
9.888 5e-001 
1.0000e+000 


-6.6403e-002 
-1.8666e-001 
-3.2061e-001 
-3.2607e-001 


-3.1022e-001 
-4.4722e-001 
-4.8968e-001 
-4.8975e-001 


- 5.916 5e-001 
-2.8896e-001 
-1.1144e-002 
-6.7428e-011 


9.7152e-001 
9.7627e-001 
9.9848e-001 
1.0000e+000 



By good fortune, our initial estimates /'(0) = — 1 and /"(0) = 1 were very close to the 
final values. 

PROBLEM SET8.1 

1. Numerical integration of the initial value problem 

/+ /-}'= 0 y( 0 ) = 0 3 / ( 0 ) = 1 

yielded y(l) = 0.741028. What is the value of /(0) that would resuit in y(l) = 1, 
assuming that y( 0 ) is unchanged? 

2. The solution of the differential equation 

/" + /' + 2/ = 6 

with the initial conditions y( 0 ) = 2 , /( 0 ) = 0 and /'( 0 ) = 1 , yielded y(l) = 
3.03765. When the solution was repeated with /'(0) = 0 (the other conditions 
being unchanged), the resuit was y(l) = 2.72318. Determine the value of/'(0) so 
that y(l) = 0 . 

3. Roughly sketch the solution of the following boundary value problems. Use the 
sketch to estimate /( 0 ) for each problem. 

(a) f = -e~y y(0) = 1 y(l) = 0.5 

(b) /' = 4y 1 2 3 y(0) = 10 /(1) = 0 

(c) f = cos(xy) y( 0 ) = 1 y(l) = 2 
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4. Using a rough sketch of the solution estimate of y(0) for the following boundary 
value problems. 

(a) f = f + xy y ( 0 ) = 0 y(l) = 2 

(b) f = yco) = 0 y(i) = 2 

(c) f = -x{y) 2 3 /( 0 ) =2 y(l) = 1 

5. Obtain a rough estimate of /'(0) for the boundary value problem 

/' + 5// = 0 

y( 0 ) = 0 j/( 0 ) = 1 y(l) = 0 

6 . Obtain rough estimates of /'(0) and /"(0) for the boundary value problem 

y <4) + 2 j/' + y 1 sin y = 0 


y(0) = y( 0 ) = 0 y(l) = 5 j/(1) = 0 

7. Obtain rough estimates of i(0) and j>(0) for the boundary value problem 

x + 2x 2 - y = 0 x( 0 ) = 1 jt(l) = 0 

y+y 2 - 2 x=l y( 0 ) = 0 y(l) = 1 

8 . ■ Solve the boundary value problem 

/ + (1 - 0.2x) y 2 = 0 y( 0 ) = 0 y(jr/ 2 ) = 1 


9. ■ Solve the boundary value problem 

/ + 2/ + 3y 2 = 0 y(0.01) = 0 y(2) =-1 

10. ■ Solve the boundary value problem 

3 /' + sin y + 1 = 0 y( 0 ) = 0 y(jr) = 0 

11. ■ Solve the boundary value problem 

/'+-/ + y =0 y( 0 . 01 ) = 1 /( 2 ) = 0 

and plot y vs. x. Warning: y changes very rapidly near x = 0. 

12. ■ Solve the boundary value problem 

f - (1 - e~ x ) y = 0 y( 0 ) = 1 y(oo) = 0 

and plot y vs. x. Hint : Replace the infinity by a finite value fi. Check your choice of 
fi by repeating the solution with 1.5 fi. If the results change, you must increase fi. 
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13. ■ Solve the boundaryvalue problem 

/' = --/ + -V + 0-K/) 3 

x x 2 

y(i) = o / (l) = 0 y(2) = l 

14. ■ Solve the boundary value problem 

/" + 4/ + 6/ = 10 
y(0) = /(0) = 0 y(3) - j/(3) = 5 

15. ■ Solve the boundary value problem 

/" + 2/ + sin y = 0 

y(— 1) = 0 yc-D = -1 /( 1 ) = 1 

16. ■ Solve the differential equation in Prob. 15 with the boundary conditions 

yC—l) = 0 y(0) = o y(D = i 

(this is a three-point boundary value problem). 

17. ■ Solve the boundary value problem 

y( 4) = -xy 2 

y(o) = 5 /(o) = o yui = o y"(D = 2 

18. ■ Solve the boundary value problem 

y (4) = -2 yf 

y(0) = y(0) = o y(4) = o /(4) = i 



A projectile of mass m in free flight experiences the aerodynamic drag force 
F d = cv 1 , where v is the velocity. The resulting equations of motion are 


X =- vx 

m 


y= ~m Vy ~ g 


1 = \/x 2 + y 2 


If the projectile hits a target 8 km away after a 10-s flight, determine the launch 
velocity vq and its angle of inclination 6. Use m= 20 kg, c = 3.2 x 10 1 kg/m and 
g= 9.80665 m/s 2 . 
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20. ■ 


n mTTTTnTTTTTTI n ,, 

L 


J r 


The simply supported beam carries a uniform load of intensity Wq and the tensile 
force N. The differential equation for the vertical displacement v can be shown 
to be 


cfiv N (fv U)q 
dx 4 EI dx 2 EI 


where EI is the bending rigidity. The boundary conditions are v = d l v/dx 2 = 0 at 
x = 0 and L. Changing the variables to § = x/ L and y = (EI j tu, L A )u Lransforms 
the problem to the dimensionless form 



Determine the maximum displacement if (a) /J = 1.65929; and (b) fi = —1.65929 
{N is compressive). 

21. ■ Solve the boundary value problem 

/' + yf = 0 y(0) = y (0) = o, /(oo) = 2 

and plot y[x) and / (x). This problem arises in determining the velocity profile of 
the boundary layer in incompressible flow (Blasius solutio n). 

8.3 Finite Difference Method 


y 



m+ 1 


/ 


X 


a 

Figure8.1. Finite difference mesh. 


X.i x 0 x, x 2 


x m-2 x m--\ x m x m + 1 

b 


















311 


8.3 Finite Difference Method 


In the finite difference method we divide the range of integration (a, b) into m 
equal subintervals of length h each, as shown in Fig. 8.1. The values of the numerical 
solution at the mesh points are denoted by yt, i = 0,1,, m; the purpose of the two 
points outside (a, b) will be explained shortly. We now make two approximations: 


1. The derivatives of y in the differential equation are replaced by the finite difference 
expressions. It is common practice to use the first Central difference approxima¬ 
tions (see Chapter 5): 


y i+ 1 - Jg-i 
Yi 2 h 


,, yi -1 - 2 yt + yt+i 
Yi 


etc. 


( 8 . 8 ) 


2. The differential equation is enforced only at the mesh points. 


As a resuit, the differential equations are replaced by m+ 1 simultaneous alge- 

braic equations, the unknowns being y,-, i = 0,1,_ m. If the differential equation is 

nonlinear, the algebraic equations will also be nonlinear and must be solved by the 
Newton-Raphson method. 

Since the truncation error in a first Central difference approximation is the 
finite difference method is not nearly as accurate as the shooting method—recall that 
the Runge-Kutta method has a truncation error of Of/z 5 ). Therefore, the convergence 
criterion specified in the Newton-Raphson method should not be too severe. 


Second-Order Differential Equation 

Consider the second-order differential equation 


/ = f(x, y , /) 


with the boundary conditions 


y(a) = a or /(a) = a 
y(b) = /3 or j /{b) = p 

Approximating the derivatives at the mesh points by finite differences, the prob- 
lem becomes 


y ( --i — 2y t + y i+1 _ ( y i+1 - y,_i 

— J l -U’ yu 


It 


2 h 


yi - y -1 

y 0 = a or ——— = a 


i = 0,1, 


m 


y m = p or 


2 h 

Tmfl — Tm-1 

2 h 


(8.9) 

(8.10a) 

(8.10b) 


Note the presence of y_i and y ,,^\, which are associated with points outside the 
solution domain ( a, b). This “spillover” can be eliminated by using the boundary 
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conditions. But before we do that, let us rewrite Eqs. (8.9) as 

y -1 - 2yo + J4 - ^ yo, yi ' ) = 0 (a) 

y/-i - 2y,- + y/+! - /z 2 / (x*, y/, y ' + ' 2fe y '~' j = 0 , i = 1 , 2 ,..., m- 1 (b) 

ym-i - 2 y m + y nH .i - /z 2 / (x m , y m , j = 0 (c) 

The boundary conditions on y are easily dealt with: Eq. (a) is simply replaced 
by y 0 — a = 0 and Eq. (c) is replaced by y m — fi = 0. If' / are prescribed, we obtain 
from Eqs. (8.10) y_i = yi — 2/to and y nif i = y, fl _i + 2 hfi, which are then substituted 
into Eqs. (a) and (c), respectively. Hence we finish up with m+ 1 equations in the 
unknowns yo, yi,..., y m : 

y 0 - a = 0 if y(a) = a 1 

-2y 0 + 2yi - fi 2 f (jc 0 , y 0 , a) - 2ha = 0 if /(a) = a J 

yi -1 - 2y + y «+1 - fc 2 / fx,-, y-, J/;+1 2 ^ J/i ~ 1 j =0 i=l,2,...,m- 

y m - /3 = 0 if y(fo) = y6 | 

2ym_i - 2y m - /t 2 f(x m y m , P) + 2hp = 0 if/(b) = /S J 


(8. lia) 

1 (8.11b) 

(8.11c) 


EXAMPLE 8.6 

Write out Eqs. (8.11) for the following linear boundary value problem using m= 10: 

/ = —4y + 4x y(0) = 0 yV/2) = 0 

Solve these equations with a computer program. 

Solution Inthiscasea = y(0) = 0, p = y{jz/2) = Oand /(x, y, /) = —4y+4x. Hence 
Eqs. (8.11) are 

yo = 0 

y;-i - 2y, + y+i - /i 2 (-4y + 4x ( -) = 0, i = 1, 2,..., 9 
2yg — 2yio — }?[— 4yio + 4xio) = 0 
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or, using matrix notation 


"1 0 


'yo ' 


'0 

1 -2 + 4/z 2 1 


yi 


4/^Xi 

1 -2 + 4/z 2 1 


yg 


4/^Xg 

2 -2 + 4/z 2 . 


-J4o. 


_ 4h 2 Xio _ 


Note that the coefficient matrix is tridiagonal, so that the equations can be 
solved efficiently by the decomposition and back substitution routines in module 
LUdecomp3, described in Section 2.4. Recalling that in LUdecomp3 the diagonals 
of the coefficient matrix are stored in vectors c, d and e, we arrive at the following 
program: 

#!/usr/bin/python 
## example8_6 

from numarray import zeros,ones,Float64,array,arange 
from LUdecomp3 import * 
from math import pi 

def equations(x,h,m): # Set up finite difference eqs. 
h2 = h*h 

d = ones((m + l))*(-2.0 + 4.0*h2) 
c = ones((m),type = Float64) 
e = ones((m),type = Float64) 
b = ones((m+1))*4.0*h2*x 
d[0] = 1.0 
e[0] = 0.0 
b[0] = 0.0 
c[m-1] = 2.0 
return c,d,e,b 

xStart =0.0 # x at left end 

xStop = pi/2.0 # x at right end 

m = 10 # Number of mesh spaces 

h = (xStop - xStart)/m 

x = arange(xStart,xStop + h,h) 

c,d,e,b = equations(x,h,m) 

c,d,e = LUdecomp3(c,d,e) 

y = LUsolve3(c,d,e,b) 
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print ’’\n 

X 

for i in range(m 

+ D: 

print ’’%14. 

5e %14.5e ’ ’ %(. 

raw_input(’’ \nPress return to 

The solution is 

X 

Y 

0.00000e+000 

0.00000e+000 

1.57080e-001 

3.14173e-001 

3.14159e-001 

6.12841e-001 

4.71239e-001 

8.82030e-001 

6.28319e-001 

1.11068e+000 

7.85398e-001 

1.29172e+000 

9.42478e-001 

1.42278e+000 

1.099 56e+000 

1.50645e+000 

1.25664e+000 

1.54995e+000 

1.413 72e+000 

1.56451e+000 

1.57080e+000 

1.56418e+000 


The exact solution of the problem is 

y= x — sin 2x 

which yields y{n/2) = tt/ 2 = 1.57080. Thus the error in the numerical solution is 
about 0.4%. More accurate results can be achieved by increasing m For example, with 
m= 100, we would get y(j r/2) = 1.57073, which is in error by only 0.0002%. 

EXAMPLE 8.7 

Solve the boundary value problem 

/ = “ST/ y(0) = 0 y( 2) = 1 

with the finite difference method. Use m= 10 and compare the output with the results 
of the shooting method in Example 8.1. 

Solution As the problem is nonlinear, Eqs. (8.11) must be solved by the Newton- 
Raphson method. The program listed below can be used as a model for other second- 
order boundary value problems. The function residual(y) returns the residuals 
of the finite difference equations, which are the left-hand sides of Eqs. (8.11). The 
differential equation y" = /(x, y, /) is defined in the function F(x,y,yPrime). In 
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this problem we chose for the initial solution y,- = 0.5 Xi, which corresponds to the 
dashed straight line shown in the rough plot of y in Example 8.1. The starting values 
of yo, yi,..., y m are specified by function startSoln(x). Note that we relaxed the 
convergence criterion in the Newton-Raphson method to 1.0 x 10 -5 , which is more 
in line with the truncation error in the finite difference method. 

#!/usr/bin/python 
## example8_7 

from numarray import zeros,Float64,array,arange 
from newtonRaphson2 import * 

def residual(y): # Residuals of finite diff. Eqs. (8.11) 

r = zeros((m + 1),type=Float64) 
r[0] = y[0] 
r [m] = y[m] - 1.0 
for i in range(l,m): 

r[i] = y[i-1] - 2.0*y[i] + y[i+l] \ 

- h*h*F(x[i],y[i],Cy[i+1] - y[i-l])/(2.0*h) 

return r 

def F(x,y,yPrime): # Differential eqn. y’’ = F(x,y,y’) 

F = -3.0*y*yPrime 
return F 

def startSoln(x): # Starting solution y(x) 
y = zeros((m + 1),type=Float64) 
for i in range(m + 1): y[i] = 0.5*x[i] 
return y 

xStart =0.0 # x at left end 

xStop =2.0 # x at right end 

m = 10 # Number of mesh intervals 

h = (xStop - xStart)/m 

x = arange(xStart,xStop + h,h) 

y = newtonRaphson2(residual,startSoln(x),1.Oe-5) 
print ’’\n x y ’ ' 

for i in range(m + 1): 

print ’’%14.5e %14.5e’’ %(x[i],y[i]) 
raw_input(’’ \nPress return to exit’’) 




316 


Two-Point Boundary Value Problems 


Here is the output from our program together with the solution obtained in 
Example 8.1. 


X 

Y 

y from Ex. 8. 

0.00000e+000 

0.00000e+000 

0.00000e+000 

2.00000e-001 

3.02404e-001 

2.94050e-001 

4.00000e-001 

5 .54503e-001 

5.41710e-001 

6.00000e-001 

7.34691e-001 

7.21875e-001 

8.00000e-001 

8.49 794e-001 

8.39446e-001 

1.00000e+000 

9.18132e-001 

9.10824e-001 

1.20000e+000 

9.56953e-001 

9.52274e-001 

1.40000e+000 

9.78457e-001 

9.75 724e-001 

1.60000e+000 

9.90201e-001 

9.88796e-001 

1.80000e+000 

9.96566e-001 

9.96023e-001 

2.00000e+000 

1.00000e+000 

1.00000e+000 


The maximum discrepancy between the Solutions is 1.8% occurring at x = 0.6. 
As the shooting method used in Example 8.1 is considerably more accurate than the 
finite difference method, the discrepancy can be attributed to truncation errors in 
the finite difference solution. This error would be acceptable in many engineering 
problems. Again, accuracy can be increased by using a finer mesh. With m= 100 we 
can reduce the error to 0.07%, but we must question whether the tenfold increase in 
computation time is really worth the extra precision. 


Fourth-Order Differential Equation 

For the sake of brevity we limit our discussion to the special case where / and /" do 
not appear explicitly in the differential equation; that is, we consider 

y (4) = /(*, y, /) 

We assume that two boundary conditions are prescribed at each end of the so¬ 
lution domain ( a , b). Problems of this form are commonly encountered in beam 
theory. 

Again we divide the solution domain into m intervals of length h each. Replacing 
the derivatives of y by finite differences at the mesh points, we get the finite difference 
equations 


yi -2 - 4y«_i + 6y; - 4y m + y i+2 
h' 



yu 


yt-i - 2y t + y i+i \ 
V ) 


( 8 . 12 ) 
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where i = 0,1,..., m. It is more revealing to write these equations as 

y- 2 - 4y_i + 6y 0 - 4y, + y 2 - h 4 f (xo, y 0 , —— + 34 j = 0 (8.13a) 

y- 1 ~ 4yo + 6yi -4y, + y 3 - h 4 f(x u y u ^ ^ + ^ ) = 0 (8.13b) 

yo - 4yi + 6y> - 4y 3 + y 4 - /i 4 / (x 2 , y 2 , —— 2 ^ 2 + J ^ j = 0 (8.13c) 


y, f! _ 3 - 4ym_ 2 + 6ym_i - 4y m + y nifi - h 4 f^x in - 1: ym_i, — 2 ^'”~ 1 + = 0 

(8.13d) 

ym- 2 4y m _i + 6y m 4y m3 _i + y/^ 2 b f I x m , y m , ^ J — 0 

(8.13e) 

We now see that there are four unknowns y_ 2 , y_i, y irH .i and y nVr2 that lie outside the 
solution domain that must be eliminated by applying the boundary conditions, a task 
that is facilitated by Table 8.1. 


Bound. cond. 

Equivalent finite difference expression 

y[a) = a 
y[a ) = c/ 
y’{a) = a 
y"[a) = a 

yo = a 

y_ i = yi — 2 ha 

y -1 = 2y 0 - yi + lia 

y ~2 = 2y i - 2yi + y 2 - lYPa 

y{b) = p 
y{b) = p 

ym = p 
yrn = p 

Ym — P 

Ymy 1 = Ym — 1 “t - 2>hfi 

Ymy 1 = 2y m Yrn — 1 ^fi 

Ymy2 — 2yrm-l 2ym-1 “1“ Ym-2 H - 27?^/3 


Table8.1 


The astute observer may notice that some combinations of boundary conditions 
will not work in eliminating the “spillover.” One such combination is clearly y(a) = o> i 
and y"'{a ) = a 2 . The other one is /(a) = a i and y"(a) = a 2 . In the context of beam 
theory, this makes sense: we can impose either a displacement y or a shear force Ely"' 
at a point, but it is impossible to enforce both of them simultaneously. Similarly, it 
makes no physical sense to prescribe both the slope / and the bending moment Eiy 1 
at the same point. 
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EXAMPLE 8.8 


r 


* I * | X 


V 


The uniform beam of length L and bendingrigidity EI is attached to rigid supports 
at both ends. The beam carries a concentrated load P at its mid-span. If we utilize 
symmetry and model only the left half of the beam, the displacement vcan be obtained 
by solving the boundary value problem 


Use the finite difference method to determine the displacement and the bending 
moment M = —EI {(ftv/dx 2 ) at the mid-span (the exactvalues are v= PL 3 /{I92EI) 
and M = PL/ 8). 

Solution By introducing the dimensionless variables 


EI 



the problem becomes 


0 

$\ =0 

t\ = ° 

(fy\ 


l?=0 


d% 3 1 


We now proceed to writing Eqs. (8.13) taking into account the boundary condi - 
tions. Referring to Table 8.1, we obtain the finite difference expressions of the bound¬ 
ary conditions at the left end as y 0 = 0 and y_i = y\. Hence Eqs. (8.13a) and (8.13b) 
become 


yo = o 

o 

4yf + 7yi -4y 2 + y 3 = 0 


(a) 


(b) 


Equation (8.13c) is 


0 

yS- 4yi + 6y> - 4y 3 + y 4 = 0 


(c) 


At the midspan the boundary conditions are equivalent to y^ = y n j_i and 


y/m-2 — ^ynPrl 4“ ym-2 2y„i_i + 2 /i^( 1 / 2 ) — ym-2 h 
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Substitution into Eqs. (8.13d) and (8.13e) yields 

ym- 3 - 4y„i_ 2 + 7 y in -i -4y m = 0 (d) 

2y in -2 8y m i T - 6y m = (e) 

The coefficient matrix of Eqs. (a)-(e) can be made symmetric by dividing Eq. (e) by 2. 
The resuit is 


'10 0 


>0 


0 

0 7-41 


yi 


0 

0-4 6-4 1 


y? 


0 

1-4 6-4 1 


ym-2 


0 

1-4 7-4 


ytn — 1 


0 

1 -4 3_ 


_ y^n 


_ 0 . 5 /z 3 _ 


The above system of equations can be solved with the decomposition and back 
substitution routines in module LUdecompS— see Section2.4. Recall that LUdecompS 
works with the vectors d, e and f that form the diagonals of the upper half of the 
matrix. The constant vector is denoted by b. The program that sets up and solves the 
equations is 

#!/usr/bin/python 
## example8_8 

from numarray import zeros,ones,Float64,array,arange 
from LUdecomp5 import * 

def equations(x,h,m): # Set up finite difference eqs. 
h4 = h* *4 

d = ones((m + l),type = Float64)*6.0 

e = ones((m),type = Float64)*(-4.0) 

f = ones((m-1),type = Float64) 

b = zeros((m+1),type=Float64) 

d[0] = 1.0 

d [ 1] = 7.0 

e[0] = 0.0 

f[0] = 0.0 

d[m-l] = 7.0 

d[m] = 3.0 

b[m] = 0.5 *h* * 3 

return d,e,f,b 


xStart 


0.0 


# x at left end 
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xStop =0.5 # x at right end 

m = 20 # Number of mesh spaces 

h = (xStop - xStart)/m 

x = arange(xStart,xStop + h,h) 

d,e,f,b = equations(x,h,m) 

d,e,f = LUdecomp5(d,e,f) 

y = LUsolve5(d,e,f,b) 

print ’’\n x y ’ ' 

for i in range(m + 1): 

print ’’%14.5e %14.5e’’ %(x[i],y[i]) 
raw_input(’’ \nPress return to exit’’) 

When we ran the program with m= 20, the last two lines of the output were 


X 

4.75000e-001 
5.00000e-001 


y 

5.19531e-003 
5.23438e-003 


Thus at the mid-span we have 

PL 3 


1-3 


PL 3 


f|x=o. 5 t = o .5 = 5.234 38 x 10 


d * 1 2 v 

dx 2 


x=0.5L 


PL 3 / 1 d 2 y 

~ET l L 2 


5 = 0.5 


PL ym-i 2y m + ymv i 


EI 


h 2 


PL (5.19531 - 2(5.23438) + 5.19531) x 10" 


EI 


0.025 2 


= -0.125 024- 


PL 
~EI 
d 2 v 

M\x=0.5L = —EI 

In comparison, the exact solution yields 


= 0.125 024 PL 


5 = 0.5 


v\x=o.5L = 5.208 33 x 10 


-3 


PL 3 

~eT 


M\ x=0 . 5L = = 0.125 000 PL 


PROBLEM SET 8.2 

Problems 1-5 Use first Central difference approximations to transform the boundary 
value problem shown into simultaneous equations Ay = b. 

1. /' = (2 + x)y, y(0) = 0, /(1) = 5. 

2. y" = y+ x 2 , y(0) = 0, y(l) = 1. 
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3. f = e~ x y, y(0) = 1, y(l) = 0. 

4. y (4) = y"-y, y(0) = 0, /(0) = 1, y(l) = 0, /(1) = -!. 

5. y (4) = —9y+x, y(0) = y"(0) = 0, /(1) = /"(1) = 0. 

Problems 6-10 Solve the given boundary value problem with the finite difference 
method using m = 20. 

6. ■ f = xy, y(l) = 1.5 y(2) = 3. 

7. ■ y" + 2/ + y = 0, y(0) = 0, y(l) = 1. Exact solution is y = xe 1_x . 

8. Wx 2 y" + x/ + y= 0, y(l) = 0, y(2) = 0.638961. Exact solution is y= sin(lnx). 

9. ■ y" = y 2 siny, /(0) = 0, y(7r) = 1. 

10. ■ y" + 2y(2x/ + y) = 0, y(0) = 1/2, /(1) =—2/9. Exact solution is 

y = (2 + x 2 ) -1 . 

11 . ■ 



X 


The simply supported beam consists of three segments with the moments of 
inertia I 0 and I\ as shown. A uniformly distributed load of intensity Wq acts over 
the middle segment. Modeling only the left half of the beam, we can show that 
the differential equation 


d 2 v M 

dx 2 EI 


for the displacement v is 


x 

d 2 v WqL 2 L 


in 0 < x < — 
4 



Introducing the dimensionless variables 


x EI 0 


h 


the differential equation changes to 


1 

m 0 < £ < - 
s 4 



2 ' 


d 2 y 

di 2 
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with the boundary conditions 


dv I 

yiiM) = ~jz 


= o 


d% U=i/2 

Use the finite difference method to determine the maximum displacement of the 
beam using m= 20 and y = 1.5 and compare it with the exact solution 

61 w 0 L i 


tVnax — 


9216 EI 0 


12 . 


Mn 


d o if^-- Id- -^ 'd. 

\/ ~ ' 


The simply supported, tapered beam has a circular cross section. A couple of 
magnitude Mq is applied to the left end of the beam. The differential equation for 
the displacement v is 


where 


Substituting 


d 2 v _ M _ Mo(l-x/L) 
dx^ ~~~EI ~ Elotd/do) 4 


d = do 




x 

~L 




y = 


E Io 
MqL 2 ‘ 


S = 


d\ 

do 


the differential equation changes to 

d 2 y = i -g 
d? ~ [1 + (5 - l)f ] 4 


with the boundary conditions 


ylf= 0 = yl f =i = o 

Solve the problem with the finite difference method using S = 1.5 and m= 20; 
plot yvs. £. The exact solution is 

(3 + 2<5§ — 3£)§ 2 £ 

y 6(l + 5§-f) 2 + 35 

13. ■ Solve Example 8.4 by the finite difference method with m= 20. Hint: Compute 
the end slopes from the second noncentral differences in Tables 5.3. 

14. ■ Solve Prob. 20 in Problem Set 8.1 with the finite difference method. Use m= 20. 
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15. ■ 


w Q 



The simply supported beam of length L is resting on an elastic foundation of 
stiffness fcN/m 2 . The displacement v of the beam due to the uniformly distributed 
load of intensity mi N/m is given by the solution of the boundary value problem 


d A v d 2 y 

EI dx* +kv= m ’ =5? 


x=0 


d 2 v 

== d? 


= 0 


x=L 


The nondimensional form of the problem is 


d A y 


d 2 y\ 


d 2 y 


*i + yy =1 ’ ^=o=^| = yit=i = 


dH 


?=o 


dx 2 


= 0 


f=i 


where 


x 

? =L 


EI 

y= ——v 


1cL a 


r = 


u\)L 4 ' EI 

Solve this problem by the finite difference method with y = 10 5 and plot y vs. £. 


16. ■ Solve Prob. 15 if the ends of the beam are free and the load is confined to the 
middle half of the beam. Consider only the left half of the beam, in which case 
the nondimensional form of the problem is 


d A y jo in 0 < § < 1/4 

+ yy — 1 1 inl / 4< | <1 / 2 
d 2 y = d 3 y I = dy\ = d 3 y\ = Q 
d % 2 || =0 d % 3 1 ?=0 d % d % 3 || = i /2 

17. ■ The general form of a linear, second-order boundary value problem is 


/' = r{x) + s(x)y+ f(x)/ 


y(a) = a or /(a) = a 
y(b) = /3 or /(b) = p 

Write a program that solves this problem with the finite difference method for 
any user-specified r(x), s(x) and Kx). Test the program by solving Prob. 8. 











Symmetric Matrix Eigenvalue Problems 


Find X for which nontrivial Solutions of Ax =Xx exist. 


9.1 Introduction 


The Standard form of the matrix eigenvalue problem is 

Ax = ax (9.1) 

where A is a given n x n matrix. The problem is to find the scalar A and the vector x. 
Rewriting Eq. (9.1) in the form 

(A — /I) x = 0 (9.2) 

it becomes apparent that we are dealing with a system of nhomogeneous equations. 
An obvious solution is the trivial one x = 0. A nontrivial solution can exist only if the 
determinant of the coefficient matrix vanishes; that is, if 


|A — AI| = 0 (9.3) 

Expansion of the determinant leads to the polynomial equation, also known as the 
characteristic equation 

Oq T- a\X -f- a%xf -\- • • • T- OnX n — 0 


which has the roots Xt, i = 1,2,..., n, called the eigenvalues of the matrix A. The 
Solutions x,- of (A — /.,■!) x = 0 are known as the eigenvectors. 

As an example, consider the matrix 



-1 

2 

-1 


0 

-1 

1 


(a) 
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The characteristic equation is 


IA-A.II 


1 — A. 

-1 

0 

-1 

2 — X 

1 

I — 1 

0 

-1 

t— 1 

1 

>-* 


= -3A + 4X 2 — X 3 = 0 


(b) 


The roots of this equation are A.i = 0, X 2 = 1, /.3 = 3. To compute the eigenvector 
corresponding the A. 3l we substitute X = /.3 into Eq. (9.2), obtaining 


‘-2 -1 o' 


Xl 


"O" 

-1 -1 -1 



= 

0 

1 

CN 

1 

i-H 

1 

O 

_ 1 


_x 3 _ 


0 


We know that the determinant of the coefficient matrix is zero, so that the equations 
are not linearly independent. Therefore, we can assign an arbitrary value to any one 
component of x and use two of the equations to compute the other two components. 
Choosing X\ = 1, the first equation of Eq. (c) yields x 2 = —2 and from the third equa¬ 
tion we get jc 3 = 1. Thus the eigenvector associated with A. 3 is 


x 3 = 


1 

-2 

1 


The other two eigenvectors 



r 


" 1 " 

x 2 = 

0 

Xl = 

1 


-1 


1 


can be obtained in the same manner. 

It is sometimes convenient to display the eigenvectors as columns of a matrix X. 
For the problem at hand, this matrix is 


X = 


[ Xl 


x 2 


X 3 ] 


1 1 1 

1 0 -2 

1 -1 1 


It is ciear from the above example that the magnitude of an eigenvector is indeter¬ 
minate; only its direction can be computed from Eq. (9.2). It is customary to normalize 
the eigenvectors by assigning a unit magnitude to each vector. Thus the normalized 
eigenvectors in our example are 



"1/V3 

1/V2 

1/V6~ 

X = 

1/V3 

0 

—2/V6 


1/V3 

-1/V2 

1/V6 


Throughout this chapter we assume that the eigenvectors are normalized. 
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Here are some useful properties of eigenvalues and eigenvectors, given without 
proof: 

• All eigenvalues of a symmetric matrix are real. 

• All eigenvalues of a symmetric, positive-definite matrix are real and positive. 

• The eigenvectors of a symmetric matrix are orthonormal; that is, X r X = I. 

• If the eigenvalues of A are A.,-, then the eigenvalues of A -1 are kj'. 

Eigenvalue problems that originate from physical problems often end up with a 
symmetric A. This is fortunate, because symmetric eigenvalue problems are easier to 
solve than their nonsymmetric counterparts (which may have complex eigenvalues). 
In this chapter we largely restrict our discussion to eigenvalues and eigenvectors of 
symmetric matrices. 

Common sources of eigenvalue problems are the analysis of vibrations and sta- 
bility. These problems often have the following characteristics: 

• The matrices are large and sparse (e.g., have a banded structure). 

• We need to know only the eigenvalues; if eigenvectors are required, only a few of 

them are of interest. 

A useful eigenvalue solver must be able to utilize these characteristics to minimize 
the computations. In particular, it should be flexible enough to compute only what 
we need and no more. 


9.2 Jacobi Method 

Jacobi method is a relatively simple iterative procedure that extracts allthe eigenvalues 
and eigenvectors of a symmetric matrix. Its utility is limited to small matrices (less 
than 20 x 20), because the computational effort increases very rapidly with the size 
of the matrix. The main strength of the method is its robustness—it seldom fails to 
deliver. 

Similarity Transformation and Diagonalization 

Consider the Standard matrix eigenvalue problem 


Ax = /,x 


(9.4) 


where A is symmetric. Let us now apply the transformation 


x = Px 


(9.5) 
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wherePisanonsingularmatrix. SubstitutingEq. (9.5) into Eq. (9.4) andpremultiplying 
each side by P -1 , we get 

P -1 APx* = AP _1 Px* 


A*x* = Ax* (9.6) 

where A* = P -1 AP. Because '/. was untouched by the transformation, the eigenvalues of 
A are also the eigenvalues of A*. Matrices that have the same eigenvalues are deemed to 
be similar, and the transformation between them is called a similarity transformation. 

Similarity transformations are frequently used to change an eigenvalue problem 
to a form that is easier to solve. Suppose that we managed by some means to find a P 
that diagonalizes A*. Equations (9.6) then are 


0 

o 


'xf 


O 

0 A% 2 — k ■ 

■ • O 


a 2 

= 

0 

0 0 

• A *nn ~ K 


1 

. . 

_1 


1 

. . O 
_1 


which have the Solutions 

M = k 2 = A^ ••• K=A* m (9.7) 



H- 1 


'0' 


'0 


0 


i 


0 

ii 

V 

1 

... O 
_1 

II 

1 

... O 
_1 

X* — 

A 77 _ 

1 

... i~H 

_1 


x * =[ X t X 2 • • • X *J = 1 

According to Eq. (9.5) the eigenvectors of A are 

X = PX* = PI = P (9.8) 

Elence the transformation matrix P contains the eigenvectors of A, and the eigenvalues 
of A are the diagonal terms of A*. 

Jacobi Rotation 

A special similarity transformation is the plane rotation 


x = Rx 


(9.9) 
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where 


k i 

"l 0 0 0 0 0 0 o' 

0 1 0 0 0 0 0 0 

00 c OOsOO 

R _00 0 10000 

“ 0 0 0 01000 
00— sOOcOO 
0 0 0 0 0 0 1 0 

0 0 0 0 0 0 0 1 

is called the Jacobi rotation matrix. Note that R is an identity matrix modified by the 
terms c = cos@ and s = sin 0 appearing at the intersections of columns/rows k and 
i, where 6 is the rotation angle. The rotation matrix has the useful property of being 
orthogonal, meaning that 

R -1 = R r (9.11) 

One consequence of orthogonality is that the transformation in Eq. (9.9) has the 
essential characteristic of a rotation: it preserves the magnitude of the vector; that is, 
|x| = |x*|. 

The similarity transformation corresponding to the plane rotation in Eq. (9.9) is 

A* = R -1 AR = R r AR (9.12) 

The matrix A* not only has the same eigenvalues as the original matrix A, but thanks 
to orthogonality of R, it is also symmetric. The transformation in Eq. (9.12) changes 
only the rows/columns k and t of A. The formulas for these changes are 

A* kk = C 2 Abfc + S 2 Ait ~ 2-CSAid 

A*p( = c 2 Au + s 2 A kk + 2 csAke 

Alt = Kk = - s 2 )A h + cs{A kk - A u ) (9.13) 

A ki = A* ik = cA ki - sAu, i^k, i^l 
Au — d — cAu + sA k i , i / k, i ^ i 

Jacobi Diagonalization 

The angle 6 in the Jacobi rotation matrix can be chosen so that A* kf = A* tk = 0. This 
suggests the following idea: why not diagonalize A by looping through ali the off- 
diagonal terms and zero them one by one? This is exactly what Jacobi diagonaliza¬ 
tion does. However, there is a major snag—the transformation that annihilates an 


(9.10) 
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off-diagonal term also undoes some of the previously created zeroes. Fortunately, it 
turns out that the off-diagonal terms that reappear will be smaller than before. Thus 
Jacobi method is an iterative procedure that repeatedly applies Jacobi rotations until 
the off-diagonal terms have virtually vanished. The final transformation matrix P is 
the accumulation of individual rotations R,: 


P = R 1 R 2 R 3 ... (9.14) 

The columns of P hnish up being the eigenvectors of A and the diagonal elements of 
A* = P 7 AP become the eigenvectors. 

Let us now look at the details of a Jacobi rotation. From Eq. (9.13) we see that 

A*u = 0if 

(c 2 — s 2 ) Aia + cs(A kk — Au) = 0 (a) 

Using the trigonometric identities c? — s 2 = cos 2 9 — sin 2 9 = cos 29 and cs = 
cos 9 sin 9 = (1/2) sin 29, we obtain from Eq. (a) 

2 Au 

tan2 9 =- (b) 

A kk — Au 

which could be solved for 9, followed by computation of c = cos 9 and 5 = sin 9 . How- 
ever, the procedure described below leads to better algorithm . 23 
Introducing the notation 

Alrlr — Aef 

0 = cot20 =- — -- (9.15) 

2 Au 

and utilizing the trigonometric identity 

2 t 

tan2 9 =-— 

(1 - fi ) 

where t = tan 9, we can write Eq. (b) as 

fi + 2<pt- 1 = 0 


which has the roots 

t — —(p =t (jfi + 1 

It has been found that the root |t| < 1, which corresponds to \9\ < 45°, leads to the 
more stable transformation. Therefore, we choose the plus sign if 0 > 0 and the minus 
sign if 0 < 0 , which is equivalent to using 

t = sgn( 0 ) ( - | 0 | + a/ 0 2 + 1 


23 The procedure is adapted from Press, W. H., etal., Numerical Recipes in Fortran, 2nd ed, Cambridge 
University Press, 1992. 
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To forestall excessive roundoff error if 0 is large, we multiply both sides of the equation 
by |0| + v ; 0 2 + 1 and solve for t, which yields 


sgn(0) 

101 + V<t> 2 + 1 


(9.16a) 


In the case of very large 0, we should replace Eq. (9.16a) by the approximation 


t = 


1 

20 


(9.16b) 


to prevent overflow in the computation of 0 2 . Having computed t, we can use the 
trigonometric relationship tan 9 = sin 6 / cos 9 = Vl — cos 2 9 / cos 9 to obtain 


1 


s = tc 


(9.17) 


We now improve the transformation formulas in Eqs. (9.13). Solving Eq. (a) for 
A f j, we obtain 


A u 


Akk + Aki 


Y-s 2 

cs 


(c) 


Replacing all occurrences of Au by Eq. (c) and simplifying, we can write the transfor¬ 
mation formulas in Eqs.(9.13) as 


A*kk = Akk ~ tAki 
Au = Au + tAki 

A* kt = A* lk = 0 (9.18) 

A* ki = A* k =A ki -s[A li + TAk i ), i±k, i^l 
Au = A*t = Au + s(A k i — x Au), i / k, i =£ l 


where 


s 

T = - 

1 + C 


(9.19) 


The introduction of r allowed us to express each formula in the form (original value) 
+ (change), which is helpful in reducing the roundoff error. 

At the start of Jacobi’s diagonalization process the transformation matrix P is 
initialized to the identity matrix. Each Jacobi’s rotation changes this matrix from P to 
P* = PR. The corresponding changes in the elements of P can be shown to be (only 
the columns k and l are affected) 


P* k = Pik-siPu + xPik ) 
P*l = Pii + S{P ik - T Pu) 


(9.20) 
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We stili have to decide the order in which the off-diagonal elements of A are to be 
eliminated. Jacobi’s original idea was to attack the largest element since this results 
in fewest number of rotations. The problem here is that A has to be searched for 
the largest element after every rotation, which is a time-consuming process. If the 
matrix is large, it is faster to sweep through it by rows or columns and annihilate 
every element above some threshold value. In the next sweep the threshold is lowered 
and the process repeated. We adopt Jacobi’s original scheme because of its simpler 
implementation. 

In summary, the Jacobi diagonalization procedure, which uses only the upper half 
of the matrix, is: 

1. Find the largest (absolute value) off-diagonal element /R t in the upper half of A. 

2. Compute 0, t, c and s from Eqs. (9.15)—(9.17). 

3. Compute r from Eq. (9.19). 

4. Modify the elements in the upper half of A according to Eqs. (9.18). 

5. Update the transformation matrix P using Eqs. (9.20). 

6 . Repeat steps 1-5 until the A& < e, where e is the error tolerance. 


■ jacobi 

This function computes all eigenvalues Aand eigenvectors x,- of a symmetric, 
n x n matrix A by the Jacobi method. The algorithm works exclusively with the upper 
triangular part of A, which is destroyed in the process. The principal diagonal of A is 
replaced by the eigenvalues, and the columns of the transformation matrix P become 
the normalized eigenvectors. 

## module jacobi 

’’’ lam,x = jacobi(a,tol = 1.0e-9). 

Solution of std. eigenvalue problem [a]{x> = lambdajx} 
by Jacobi’s method. Returns eigenvalues in vector {lam} 
and the eigenvectors as columns of matrix [x]. 

from numarray import array,identity,diagonal 
from math import sqrt 

def jacobi(a,tol = 1.0e-9): 

def maxElem(a): # Find largest off-diag. element a[k,l] 
n = len(a) 
aMax = 0.0 
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for i in range(n-l): 

for j in range(i+1,n): 

if abs(a[i,j]) >= aMax: 
aMax = abs(a[i,j]) 
k = i; 1 = j 
return aMax,k,l 

def rotate(a,p,k,1): # Rotate to make a[k,l] = 0 
n = len(a) 

aDiff = a[l,1] - a[k,k] 

if abs(a[k,l]) < abs(aDiff)*1.Oe-36: t = a[k,l]/aDiff 
else : 

phi = aDiff/(2. 0*a[k, 1] ) 
t = 1.0/(abs(phi) + sqrt(phi**2 + 1.0)) 
if phi < 0.0: t = -t 
c = 1.0/sqrt(t**2 + 1.0); s = t*c 
tau = s/(1.0 + c) 
temp = a[k,l] 
a[k,1] = 0.0 

a[k,k] = a[k,k] - t*temp 
a[l,l] = a[l,l] + t*temp 
for i in range(k): # Case of i < k 

temp = a[i,k] 

a[i,k] = temp - s*(a[i,l] + tau*temp) 
a[i,l] = a[i,l] + s*(temp - tau*a[i,l]) 
for i in range(k+1,1): # Case of k < i < 1 

temp = a[k,i] 

a[k,i] = temp - s*(a[i,l] + tau*a[k,i]) 
a[i,l] = a[i,l] + s*(temp - tau*a[i,l]) 
for i in range(1+1,n): # Case of i > 1 

temp = a[k,i] 

a[k,i] = temp - s*(a[l,i] + tau*temp) 
a[l,i] = a[l,i] + s*(temp - tau*a[l,i]) 
for i in range(n): # Update transformation matrix 

temp = p[i,k] 

p[i,k] = temp - s*(p[i,l] + tau*p[i,k]) 
p[i,l] = p[i,l] + s*(temp - tau*p[i,l]) 

n = len(a) 

maxRot = 5*(n**2) # Set limit on number of rotations 
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p = identity(n)*1.0 # Initialize transformation matrix 

for i in range(maxRot): # Jacobi rotation loop 
aMax,k,l = maxElem(a) 
if aMax < tol: return diagonal(a),p 
rotate(a,p,k,1) 

print 'Jacobi method did not converge' 

■ sortJacobi 

The eigenvalues/eigenvectors returned by jacobi are not ordered. The function listed 
below can be used to sort the eigenvalues and eigenvectors into ascending order of 
eigenvalues. 

## module sortJacobi 
’’’ sortJacobiflam,x). 

Sorts the eigenvalues {lam> and eigenvectors [x] 
in order of ascending eigenvalues. 

import swap 

def sortJacobiflam,x): 
n = len(lam) 
for i in range(n-l): 
index = i 
val = lam[i] 
for j in range(i+1,n): 
if lam[j] < val: 
index = j 
val = lam[j] 
if index != i: 

swap.swapRows(lam,i,index) 
swap.swapCols(x,i,index) 

Transformation to Standard Form 

Physical problems often give rise to eigenvalue problems of the form 

Ax = ABx (9.21) 

where A and B are symmetric nx n matrices. We assume that B is also positive defi¬ 
nite. Such problems must be transformed into the Standard form before they can be 
solved by Jacobi diagonalization. 
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As B is symmetric and positive definite, we can apply Choleski decomposition 
B = LL r , where L is a lower-triangular matrix (see Section 2.3). Then we introduce the 
transformation 


x = (L l ) T z 


(9.22) 


Substituting into Eq. (9.21), we get 


A(L _1 ) r z =kLL r (L -1 ) r z 


Premultiplying both sides by L 1 results in 

L -1 A(L -1 ) r z = A.L _1 LL r (L _1 ) r z 

Because L _1 L = L r (L _1 ) r = I, the last equation reduces to the Standard form 


Hz = kz 


(9.23) 


where 


H = L -1 A(L -1 ) 3 


(9.24) 


An important property of this transformation is that it does not destroy the symmetry 
of the matrix; i.e., a symmetric A results in a symmetric H. 

Here is the general procedure for solving eigenvalue problems of the form Ax = 

kBx: 


1 . 

2 . 

3. 

4. 

5. 


Use Choleski decomposition B = LL r to compute L. 

Compute L -1 (a triangular matrix can be inverted with relatively small computa- 
tional effort). 

Compute H from Eq. (9.24). 

Solve the Standard eigenvalue problem Hz = kz (e.g., using the Jacobi method). 
Recover the eigenvectors of the original problem from Eq. (9.22): x= (L -1 ) r z. 
Note that the eigenvalues were untouched by the transformation. 


An important special case is where B is a diagonal matrix: 


B = 


7*i o 

o p 2 


(9.25) 


0 0 ••• p n _ 
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Here 



[p\ /2 

0 

0 


\Pi 1/2 

0 

0 

L = 

0 

fiT • 

• 0 

L" 1 = 

0 

p- 2 1/2 • 

0 


0 

0 • 

• P 1 ' 2 . 


0 

0 

• Pn 1 ' 2 . 


(9.26a) 


and 


Au 

Ha = 11 


JMj 


(9.26b) 


■ stdForm 

Given the matrices A and B, the function stdForm returns H and the transformation 
matrix T = (L -1 ) T . The inversion of L is carried out by invert (the triangular shape 
of L allows this to be done by back substitution). Note that original A, B and L are 
destroyed. 

## module stdForm 

’’’ h,t = stdForm(a,b). 

Transforms the eigenvalue problem [a]{x> = lambda[b]{x> 
to the Standard form [h]{z> = lambda{z>. The eigenvectors 
are related by {x} = [t]{z> . 


from numarray import dot,matrixmultiply,transpose 
from choleski import * 


def stdForm(a,b): 


def invert(L): # Inverts lower triangular matrix L 
n = len(L) 

for j in range(n-l): 

L[j , j] = 1.0/L[j,j] 
for i in range(j+l,n): 

L[i,j] = -dot(L[i,j:i],L[j:i,j])/L[i,i] 
L[n-l,n-l] = 1.0/L[n-l,n-1] 


n = len(a) 

L = choleski(b) 
invert(L) 

h = matrixmultiply(b,matrixmultiply(a,transpose(L))) 
return h,transpose(L) 
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EXAMPLE 9.1 



The stress matrix (tensor) corresponding to the state of stress shown is 


S = 


80 

30 

0 


30 

40 

0 


0 

0 

60 


MPa 


(each row of the matrix consists of the three stress components acting on a coordinate 
plane). It can be shown that the eigenvalues of S are the principal stresses and the 
eigenvectors are normal to the principal planes. (1) Determine the principal stresses 
by diagonalizing S with one Jacobi rotation and (2) compute the eigenvectors. 


Solution of Part(1) To eliminate S 12 we must apply a rotation in the 1-2 plane. With 
k= 1 andf = 2Eq. (9.15) is 

Sn — S 22 80 — 40 2 

^ = 2 S 12 = 2(30) = _ 3 

Equation (9.16a) then yields 


sgn (</>) 

\<t>\ + V<P 2 +1 


- , = -0.535 18 

2/3 + ,/ (2/3) 2 + 1 


According to Eqs. (9.18), the changes in S due to the rotation are 


SJ/ = Sn - iSn = 80 - (-0.535 18) (30) = 96.055 MPa 
S( 2 = S >2 + tS l2 = 40 + (-0.535 18) (30) = 23.945 MPa 
^12 = 3>1 = 0 


Hence the diagonalized stress matrix is 


S* 


96.055 0 0 

0 23.945 0 

0 0 60 


where the diagonal terms are the principal stresses. 
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Solution of Part (2) To compute the eigenvectors, westartwithEqs. (9.17) and (9.19), 
which yield 


1 


c = 


s/TTf 


= 0.88168 


'1 + (-0.535 18) z 
s= tc= (-0.535 18) (0.881 68) = -0.471 86 
s -0.47186 

= -0.25077 


1 + c 1+0.88168 
We obtain the changes in the transformation matrix P from Eqs. (9.20). Because 
P is initialized to the identity matrix, the first equation gives us 


P*i = Ai — s(P\2 + r Ai) 

= 1 - (-0.471 86) [0 + (-0.250 77) (1)] = 0.881 67 
P 2 i = Ai - s{P 22 + rP 21 ) 

= 0 - (-0.471 86) [1 + (-0.250 77) (0)] = 0.471 86 


Similarly, the second equation of Eqs. (9.20) yields 

P* 2 = -0.471 86 P 2 * 2 = 0.881 67 

The third row and column of P are not affected by the transformation. Thus 


p* 


0.88167 -0.47186 0 
0.47186 0.88167 0 

0 0 1 


The columns of P* are the eigenvectors of S. 


EXAMPLE 9.2 


L L 2L 


v.QQQQQy 

- MStib - 

-,00000,— 

■ co 

2 C 

_(^3 

hj 




(1) Show that the analysis of the electric Circuit shown leads to a matrix eigenvalue 
problem. (2) Determine the circular frequencies and the relative amplitudes of the 
currents. 


Solution of Part (1) Kirchoffs equations for the three loops are 

^di\ qi - q 2 _ 
dt + 3 C 

, di 2 | q 2 - c/, | q 2 - q 2 __ n 

dt + 3C + C 

(I ii (fi — q 2 (/; 

2 L—- + — - - + — = o 

dt C C 
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Differentiating and substituting dq k /dl = i k , we get 

1 . 1 . 

1 1 — l 2 — LC T-7T- 

3 3 dt 2 

14 (fi 2 

~3 h + 3 l2 ~ l3 = ~ LC ~df 


— i 2 2. i 2 — — 2 LC 
These equations admit the solution 

4(f) = u k sinent 


<&h 

dt 2 


where u> is the circular frequency of oscillation (measured in rad/s) and 14 are the 
relative amplitudes of the currents. Substitution into Kirchoffs equations yields Au = 
ABu (siniwt cancels out), where 


1/3 

-1/3 

o‘ 


"l 

0 

0 “ 

-1/3 

4/3 

-1 

B = 

0 

1 

0 

0 

-1 

2 


0 

0 

2 


which represents an eigenvalue problem of the nonstandard form. 


Sol ution of Pa rt (2) Since B is a diagonal matrix, we can readily transform the problem 
into the Standard form Hz = kz. From Eq. (9.26a) we get 


and Eq. (9.26b) yields 


L 1 


1 0 0 
0 1 0 
0 0 1 /V 2 


H = 


1/3 

-1/3 

0 


-1/3 0 

4/3 -1/V2 
-1/V2 1 


The eigenvalues and eigenvectors of H can now be obtained with the Jacobi method. 
Skipping the details, we obtain the following results: 


/ 1 = 0.14779 k 2 = 0.58235 ^ = 1.93653 



~ 0.810 27" 


0.562 74’ 


0.163 70’ 

Zi = 

0.45102 

z 2 = 

-0.42040 

Z 3 = 

-0.787 30 


0.374 23 


-0.71176 


0.594 44 
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The eigenvectors of the original problem are recovered from Eq. (9.22): y,= (L 1 ) 7 z,, 
which yields 



" 0 . 81027 " 


0.562 74" 


0.163 70“ 

Ui = 

0.45102 

u 2 = 

-0.420 40 

u 3 = 

-0.787 30 


0.264 62 


-0.503 29 


0.420 33 


These vectors should now be normalized (each z,• was normalized, but the transfor- 
mation to u, does not preserve the magnitudes of vectors). The circular frequencies 
are o>, = i/ (LC) , so that 

0.3844 0.7631 1.3916 

CO\ — . (i) 2 = , CO 3 = , 

yic AIc AIc 

EXAMPLE 9.3 

-10 12 n-1 n n mn+2 

P — II II J |" H- X 

The propped cantilever beam carries a compressive axial load P. The lateral displace- 
ment u{x) of the beam can be shown to satisfy the differential equation 

u (4) + — il' = 0 (a) 

EI 

where EI is the bending rigidity. The boundary conditions are 

u{ 0) = il\ 0) = 0 u{L ) = ii{L) = 0 (b) 

(1) Show that displacement analysis of the beam results in a matrix eigenvalue problem 
if the derivatives are approximated by finite differences. (2) Use the Jacobi method to 
compute the lowest three buckling loads and the corresponding eigenvectors. 

Solution of Part (1) We divide the beaminto n+ 1 segments oflength L/[n+ 1) each 
as shown. Replacing the derivatives of uin Eq. (a) by Central finite differences of df/t 2 ) 
at the interior nodes (nodes 1 to n), we obtain 

Ui -2 - 4Ui_! + 6Ui - 4 Ui+i + u i+ 2 
/t 1 

P -Ui-Y + Zut - m+i . 

= --, 1 = 1,2,..., n 

EI W 

After multiplication by H\ the equations become 


U- 1 - 4 uq + 6Ut - 4 u 2 + 113= X[-Uo + 2ui - u 2 ) 
Uq — 4ui + 6m 2 — 4t^ + U 4 = X[—Ui + 2 u 2 — U 3 ) 


(c) 
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u n - 3 - 4 Un —2 + 6w„_i - 4 U„ + U n+ i = X{-U n - 2 + 2l/„_i - Un) 
Un—2 4 Ufi—i 4“ 6 u n 4//„+1 + U/i+2 — C Un—i 4~ 2 ti n i) 


where 


P/z 2 PL 2 
~ ~eT ~ ( n+ 1 ) 2 EI 

The displacements zz_ 1 , i<o, u„ + \ and n„ +2 can be eliminated by using the prescribed 
boundary conditions. Referring to Table 8.1, we obtain the finite difference approxi- 
mations to the boundary conditions: 


Uo = 0 n_ 1 = — U\ Hn + 1 = 0 U n +2 = Un 


Substitution into Eqs. (c) yields the matrix eigenvalue problem Ax = ABx, where 


5-4 1 0 0 ••• 0' 

-4 6-4 1 0 ••• 0 

1-4 6-4 1 • • • 0 


A = 


0 

0 

0 

2 

-1 

0 


1 

0 

0 

-1 0 

2 -1 

-1 2 


-4 6 

1 -4 

0 1 

0 0 
0 0 
-1 0 


-4 1 

6 -4 
-4 7_ 

0' 

0 

0 


B = 


0 

0 

0 


0-1 2-1 0 
0 0-1 2-1 
000-12 


Sol ution of Part (2) The problem with the Jacobi method is that it insists on finding all 
the eigenvalues and eigenvectors. It is also incapable of exploiting banded structures 
of matrices. Thus the program listed below does much more work than necessary for 
the problem at hand. More efficient methods of solution will be introduced later in 
this chapter. 


#!/usr/bin/python 
## example9_3 

from numarray import array,zeros,Float64 
from stdForm import * 
from jacobi import * 
from sortJacobi import * 
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n = 10 

a = zeros((n,n),type=Float64) 
b = zeros((n,n),type=Float64) 
for i in range(n): 
a[i,i] = 6.0 
b[i,i] = 2.0 
a[0,0 ] = 5.0 
a[n-l,n-l] = 7.0 
for i in range(n-l): 
a[i,i+l] = -4.0 
a[i+1,i] = -4.0 
b[i,i+l] = -1.0 
b[i+1,i] = -1.0 
for i in range(n-2): 
a[i,i+2] = 1.0 
a[i+2,i] = 1.0 


h,t = stdForm(a,b) 
lam,z = jacobi(h) 
x = matrixmultiply(t,z) 
for i in range(n): 

xMag = sqrt(dot(x[: 


# Convert to std. form 

# Solve by Jacobi mthd. 

# Eigenvectors of orig. prob. 

# Normalize eigenvectors 
i],x[:,i])) 


x[:,i] = x[:,i]/xMag 

sortJacobi(lam,x) # Arrange in ascending order 

print '’Eigenvalues:\n'’,lam[0:3] 
print '’ \nEigenvectors :\n'',x[:,0:3] 
raw_input('’\n Press return to exit’’) 


Running the program with n = 10 resulted in the following output: 


Eigenvalues : 

[ 0.16410379 

Eigenvectors : 

[[ 0.16410119 
[ 0.30618978 
[ 0.40786549 
[ 0.45735999 
[ 0.45146805 
[ 0.39607358 
[ 0.30518404 


0.47195675 


-0.18476623 
-0.26819121 
-0.19676237 
0.00994855 
0.26852252 
0.4710634 
0.53612023 


0.90220118] 


0.30699491] 
0.36404289] 
0.14669942] 
-0.12192373] 
-0.1724502 ] 
0.06772929] 
0.40894875] 




342 


Symmetric Matrix Eigenvalue Problems 


[ 0.19863178 0.44712859 0 . 57038382 ] 

[ 0.09881943 0.26022826 0 . 43341183 ] 

[ 0.0270436 0.07776771 0.1486333 ]] 

The first three mode shapes, which represent the relative displacements of the 
buckled beam, are plotted below (we appended the zero end displacements to the 
eigenvectors before plotting the points). 


0.6 



0.4 


0.2 


u 


0.0 


- 0.2 


-0.4 


The buckling loads are given by P, = (n + l) 2 a/ HI/L 2 . Thus 


(ll) 2 (0.164103 7) EI 

T 2 

(ll) 2 (0.471956 75) EI 
L 2 

(ll) 2 (0.902 201 18) EI 
L 2 



Pi 




The analytical values are P\ = 20.19 EI/L 2 , P 2 = 59.68 EI/L 2 and P 3 = 118.9P//L 2 . It 
can be seen that the error introduced by the finite difference approximation increases 
with the mode number (the error in P,- + i is larger than in p). Of course, the accuracy 
of the finite difference model can be improved by using larger n, but beyond n = 20 
the cost of computation with the Jacobi method becomes rather high. 

9.3 Inverse Power and Power Methods 
Inverse Power Method 

The inverse power method is a simple and efficient algorithm that finds the smallest 
eigenvalue 7, i and the corresponding eigenvector xi of 


Ax = 7,x 


(9.27) 
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The method works like this: 

1. Let v be an approximation to xi (a random vector of unit magnitude will do). 

2. Solve 


Az = v 


(9.28) 


for the vector z. 

3. Compute|z|. 

4. Let v = z/|z| and repeat steps 2-4 until the change in v is negligible. 

At the conclusion of the procedure, |z| = ±1/Ai and v = xi. The sign of Ai is de- 
termined as follows: if z changes sign between successive iterations, A i is negative; 
otherwise, Ai is positive. 

Let us now investigate why the method works. Since the eigenvectors x, of 
Eq. (9.27) are orthonormal (linearly independent), they can be used as the basis for 
any n-dimensional vector. Thus v and z admit the unique representations 


n n 


v=£ ViXi Z=£ Z{Xi 


(a) 


i= 1 i= 1 


where Vi and z,- are the components of v and z with respect to the eigenvectors x/. 
Substitution into Eq. (9.28) yields 


n n 


A Y ZiXi - Y v i*i = 0 


i= 1 i= 1 


But Ax/ = kiXi t so that 


n 


Y ( Z i X i ~ v i) x i = 0 


Hence 



It follows from Eq. (a) that 




(9.29) 


Since Ai/A, < 1 (i / 1), we observe that the coefficient of Xi has become more promi¬ 
nent in z than it was in v; hence z is a better approximation to xi. This completes the 
first iterative cycle. 
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In subsequent cycles we set v = z/|z| and repeat the process. Each iteration will 
increase the dominance of the first term in Eq. (9.29) so that the process converges to 

1 1 

Z = — ViXi = — Xi 
Ai Ai 

(at this stage V\ = 1 since v = xi, so that v\ = 1 , = V3 = • • • = 0 ). 

The inverse power method also works with the nonstandard eigenvalue problem 

Ax = ABx (9.30) 

provided that Eq. (9.28) is replaced by 

Az = Bv (9.31) 

The alternative is, of course, to transform the problem to Standard form before apply- 
ing the power method. 

Eigenvalue Shifting 

By inspection of Eq. (9.29) we see that the speed of convergence is determined by the 
strength of the inequality |A| /a 2 | < 1 (the second term in the equation). If |A 2 | is well 
separated from | Ai |, the inequality is strong and the convergence is rapid. On the other 
hand, close proximity of these two eigenvalues results in very slow convergence. 

The rate of convergence canbe improvedbyatechnique called eigenvalue shifting. 
If we let 


A = A* + 5 (9.32) 

where s is a predetermined “shift,” the eigenvalue problem in Eq. (9.27) is trans- 
formed to 


Ax = (A* + s)x 


or 


A*x = A*x 


where 


(9.33) 


A* = A - si (9.34) 

Solving the transformed problem in Eq. (9.33) by the inverse power method yields Aj 
and xi, where Aj is the smallest eigenvalue of A*. The corresponding eigenvalue of the 
original problem, A = Aj + s, is thus the eigenvalue closest to s. 

Eigenvalue shifting has two applications. An obvious one is the determination of 
the eigenvalue closest to a certain value 5 . For example, if the working speed of a shaft 
is 5 rpm, it is imperative to assure that there are no natural frequencies (which are 
related to the eigenvalues) close to that speed. 
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Eigenvalue shifting is also be used to speed up convergence. Suppose that we are 
computing the smallest eigenvalue A i of the matrix A. The idea is to introduce a shift 
s that makes A*/A| as small as possible. Since AJ = Ai — s, we should choose s « Ai 
(s = Ai should be avoided to prevent division by zero). Of course, this method works 
only if we have a prior estimate of Ai. 

The inverse power method with eigenvalue shifting is a particularly powerful tool 
for finding eigenvectors if the eigenvalues are known. By shifting very close to an 
eigenvalue, the corresponding eigenvector can be computed in one or two iterations. 

Power Method 

The power method converges to the eigenvalue farthest from zero and the associated 
eigenvector. It is very similar to the inverse power method; the only difference be- 
tween the two methods is the interchange of v and z in Eq. (9.28). The outline of the 
procedure is: 

1. Let v be an approximation to xi (a random vector of unit magnitude will do). 

2. Compute the vector 

z = Av (9.35) 

3. Compute \z\. 

4. Let v = z/|z| and repeat steps 2-4 until the change in v is negligible. 

At the conclusion of the procedure, |z|' = ±A„ and v = x„ (the sign of A„ is deter- 
mined in the same way as in the inverse power method). 

■ inversePower 

Given the matrix A and the shift s, the function inversePower returns the eigen¬ 
value of A closest to s and the corresponding eigenvector. The matrix A* = A — si is 
decomposed as soon as it is formed, so that only the solution phase (forward and 
back substitution) is needed in the iterative loop. If A is banded, the efficiency of the 
program could be improved by replacing LUdecomp and LUsolve by functions that 
specialize in banded matrices (e.g., LUdecomp 5 and LUsolve 5)—see Example 9.6. The 
program line that forms A* must also be modified to be compatible with the storage 
scheme used for A. 

## module inversePower 

’’’ lam,x = inversePower(a,s,tol=l.Oe-6). 

Inverse power method for solving the eigenvalue problem 
[a]{x> = lam{x}. Returns ’lam’ closest to ’s’ and the 
corresponding eigenvector {x>. 
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from numarray import zeros,Float64,dot,identity 
from LUdecomp import * 
from math import sqrt 
from random import random 


def inversePower(a,s,tol=1.0e-6) : 


n = len(a) 

aStar = a - identity(n)*s # 

aStar = LUdecomp(aStar) # 

x = zeros((n),type=Float64) 
for i in range(n): # 

x[i] = randomO 
xMag = sqrt(dot(x,x)) # 

x =x/xMag 

for i in range(50): # 

xOld = x.copyO # 

x = LUsolve(aStar,x) # 

xMag = sqrt(dot(x,x)) # 

x = x/xMag 

if dot(x01d,x) <0.0: # 

sign = -1.0 
x = -x 

else: sign = 1.0 


if sqrt(dot(xOld - x,x01d 
return s + sign/xMag,: 
print ’Inverse power method d: 


Form [a*] = [a] - s[I] 
Decompose [a*] 

Seed [x] with random numbers 

Normalize [x] 

Begin iterations 
Save current [x] 

Solve [a*][x] = [xOld] 
Normalize [x] 

Detect change in sign of [x] 


- x)) < tol: 
d not converge’ 


EXAMPLE 9.4 

The stress matrix describing the state of stress at a point is 


-30 

10 

20 

10 

40 

-50 

20 

-50 

-10 


Determine the largest principal stress (the eigenvalue of S farthest from zero) by the 
power method. 


Solution First iteration: 

Let v = 1 0 0 be the initial guess for the eigenvector. Then 


—30 10 20~ 


"l" 


” —30.0 _ 

10 40 -50 


0 

= 

10.0 

20 -50 -10 


0 


20.0 


z = Sv = 
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|z| = \/ 30 2 + 10 2 + 20 2 = 37.417 


~ —30.0’ 


"-0.801 77“ 

10.0 


0.267 26 


37.417 


20.0 J 


0.534 52 


Second iteration: 


—30 10 20“ 


’-0.801 77’ 


37.416’ 

10 40 -50 


0.267 26 

= 

-24.053 

20 -50 -10 


0.534 52 


-34.744 


jzj = x/37.416 2 + 24.053 2 + 34.744 2 = 56. 442 


37.416’ 


0.66291’ 

-24.053 


-0.42615 

56. 442 

-34.744 


-0.61557 



Third iteration: 


—30 10 20’ 


0.66291’ 


’-36.460 ’ 

10 40 -50 


-0.42615 

= 

20.362 

20 -50 -10 


-0.61557 


40.721 


\z\ = V36.460 2 + 20.362 2 + 40.721 2 = 58.328 


‘-36.460’ 

1 

’-0.62509’ 

20.362 


0.34909 


58.328 


40.721 


0.69814 



At this point the approximation of the eigenvalue we seek is X = —58.328 MPa (the 
negative sign is determined by the sign reversal of z between iterations). This is actually 
close to the second-largest eigenvalue X 2 = —58.39 MPa! By continuing the iterative 
process we would eventually end up with the largest eigenvalue /. 3 = 70.94 MPa. But 
since \X 2 \ and |/. 3 | are rather close, the convergence is too slow from this point on for 
manual labor. Here is a program that does the calculations for us: 


#!/usr/bin/python 
## example9_4 

from numarray import array,matrixmultiply,dot 
from math import sqrt 


s = arrayC[[-30.0 , 10.0, 20.0], \ 

[ 10.0, 40.0, -50.0], \ 

[ 20.0, -50.0, -10.0]]) 
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v = array([1.0, 0.0, 0.0]) 
for i in range(lOO): 
vOld = v.copyO 
z = matrixmultiplyCs,v) 
zMag = sqrt(dot(z,z)) 
v = z/zMag 

if dot(v01d,v) < 0.0: 
sign = -1.0 
v = -v 

else: sign = 1.0 

if sqrt(dot(vOld - v,v01d - v)) < 1.0e-6: break 
lam = sign*zMag 

print ’'Number of iterations = ’’,i 
print ’'Eigenvalue =’’,lam 

raw_input(’’ \nPrint press return to exit’’) 

The results are: 


Number of iterations = 92 
Eigenvalue = 70.9434833068 


Note that it took 92 iterations to reach convergence! 

EXAMPLE 9.5 

Determine the smallest eigenvalue 7. i and the corresponding eigenvector of 


A = 


'11 

2 

3 

1 

4 


2 

9 

3 

5 

2 


3 

3 

15 

4 
3 


1 4' 

5 2 

4 3 

12 4 

4 17 


Use the inverse power method with eigenvalue shifting knowing that 7. i « 5. 

Solution 

#!/usr/bin/python## 
example9_ 5 

from numarray import array 
from inversePower import * 


s 


5.0 
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arrayC[[ 

11.0, 

2.0, 

3.0, 

1.0, 

4.0] , 

\ 

[ 

2.0, 

9.0, 

3.0, 

5.0, 

2.0] , 

\ 

[ 

3.0, 

3.0, 

15.0, 

4.0, 

3.0] , 

\ 

[ 

1.0, 

5.0, 

4.0, 

12.0, 

4.0] , 

\ 

[ 

4.0, 

2.0, 

3.0, 

4.0, 

17.0]]) 



lam,x = inversePower(a,s) 

print ’'Eigenvalue =’’,lam 

print ’’ \nEigenvector :\n’’,x 

raw_input(’’ \nPrint press return to exit’’) 

Here is the output: 

Eigenvalue = 4.87394637865 


Eigenvector: 

[-0.26726603 0.74142854 0.05017271 -0.59491453 0.14970633] 

Convergence was achieved with 4 iterations. Without the eigenvalue shift 26 iter- 
ations would be required. 

EXAMPLE 9.6 

Unlike Jacobi diagonalization, the inverse power method lends itself to eigenvalue 
problems of banded matrices. Write a program that computes the smallest buckling 
load of the beam described in Example 9.3, making full use of the banded forms. Run 
the program with 100 interior nodes (n = 100). 

Solution The function inversePower5 listed below returns the smallest eigenvalue 
and the corresponding eigenvector of Ax = ABx, where A is a pentadiagonal matrix 
and B is a sparse matrix (in this problem it is tridiagonal). The matrix A is input 
by its diagonals d, e and f as was done in Section 2.4 in conjunction with the LU 
decomposition. The algorithm for inversePowerS does not use B directly, but calls 
the function Bv(v) that supplies the product Bv. Eigenvalue shifting is not used. 


## module inversePower5 

’’’ lam,x = inversePower5(Bv,d,e,f,tol=l.Oe-6). 

Inverse power method for solving the eigenvalue problem 
[A]{x> = lam[B]{x}, where [A] = [f\e\d\e\f] is 
pentadiagonal and [B] is sparse.. User must supply the 
function Bv(v) that returns the vector [B]{v>. 
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from numarray import zeros,Float64,dot 
from LUdecomp5 import * 
from math import sqrt 
from random import random 

def inversePower5(Bv,d,e,f,tol=1.0e-6) : 
n = len(d) 

d,e,f = LUdecomp5(d,e,f) 
x = zeros((n),type=Float64) 
for i in range(n): # 

x[i] = randomO 
xMag = sqrt(dot(x,x)) # 

x = x/xMag 

for i in range(30): # 

xOld = x.copyO # 

x = Bv(xOld) # 

x = LUsolve5(d,e,f,x) # 

xMag = sqrt(dot(x,x)) # 

x = x/xMag 

if dot(x01d,x) <0.0: # 

sign = -1.0 
x = -x 

else: sign = 1.0 
if sqrt(dot(xOld - x,x01d 
return sign/xMag,x 
print ’Inverse power method did not converge’ 

The prograrn that Utilizes inversePower 5 is 

#!/usr/bin/python 
## example9_6 
from numarray import ones 
from inversePower5 import * 

def Bv(v): # Computes {z} 

n = len(v) 

z = zeros((n),type=Float64) 
z[0] = 2.0*v[0] - v[l] 
for i in range(1,n-1): 

z[i] = -v[i-l] + 2.0*v[i] 


= [B]{v> 


- v[i+l] 


Seed {v> with random numbers 

Normalize {v} 

Begin iterations 
Save current {v} 

Compute [B]{v> 

Solve [A]{z> = [B]{v> 
Normalize {z} 

Detect change in sign of {x} 


- x)) < tol: 
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z[n-l] = -v[n-2] + 2.0*v[n-l] 
return z 


n = 100 # Number of interior nodes 

d = ones((n))*6.0 # Specify diagonals of [A] = [f\e\d\e\f] 

d[0] = 5.0 

d[n-l] = 7.0 

e = ones((n-1))*(-4.0) 

f = ones((n-2))* 1.0 

lam,x = inversePower5(Bv,d,e,f) 

print ’’ PL~ 2/EI =” ,lam*(n+l)**2 

raw_input(’’ \nPress return to exit’’) 

The output, shown below, is in excellent agreement with the analytical value. 


PL'2/EI = 20.1867355603 

PROBLEM SET 9.1 


1. Given 


"7 

3 

1 


'4 

0 

o' 

3 

9 

6 

B = 

0 

9 

0 

1 

6 

8 


0 

0 

4 


A = 


convert the eigenvalue problem Ax = 7.Bx to the Standard form Hz = Az. What is 
the relationship between x and z? 

Convert the eigenvalue problem Ax = itBx, where 


A = 


to the Standard form. 

3. An eigenvalue of the problem in Prob. 2 is roughly 2.5. Use the inverse power 

method with eigenvalue shifting to compute this eigenvalue to four decimal 
places. Start withx= |^1 0 oj . Hin!: two ilerations sliould be sufficient. 

4. The stress matrix at a point is 


4 

-1 

o ’ 


2 

-1 

o " 

-1 

4 

-1 

B = 

-1 

2 

-1 

0 

-1 

4 


0 

-1 

1 


S = 


150 -60 0 

-60 120 0 
0 0 80 


MPa 


Compute the principal stresses (eigenvalues of S). 
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5. 



2 m 


The two pendulums are connected by a spring which is undeformed when the 
pendulums are vertical. The equations of motion of the system can be shown 
to be 


kL[6 2 — 0\) — mgdi = mL6\ 

—kL(6 2 — Oi ) — 2 mgd 2 = 2 mL6 2 

where 6\ and 0 2 are the angular displacements and k is the spring stiffness. 
Determine the circular frequencies of vibration and the relative amplitudes 
of the angular displacements. Use m= 0.25 kg, k= 20 N/m, L = 0.75 m and 
g= 9.80665 m/s 2 . 


L L 



Kirchoff’s laws for the electric Circuit are 


cf i\ 

3 1\ — I 2 — Z 3 = — LC 


— h + h 
—ii + h 


= -LC 


= -LC 


dfi 

cfi 2 

~dfi 

(Ph 

~dfi 


Compute the circular frequencies of the Circuit and the relative amplitudes of the 
loop currents. 
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7. Compute the matrix A* that results from annihilation of A u and A, u in the matrix 

'4-1 0 1" 

-1 6-20 

A = 

0-2 3 2 

10 2 4 


by a Jacobi rotation. 

8 . ■ Use the Jacobi method to determine the eigenvalues and eigenvectors of 


A = 


4 

-1 

-2 


-1 

3 

3 


-2 

3 

1 


9. ■ Find the eigenvalues and eigenvectors of 


A = 


4 

-2 

1 

-1 


-2 

4 

-2 

1 


1 

-2 

4 

-2 


-1 

1 

-2 

4 


with the Jacobi method. 


10. ■ Use the power method to compute the largest eigenvalue and the corresponding 
eigenvector of the matrix A given in Prob. 9. 

11. ■ Find the smallest eigenvalue and the corresponding eigenvector of the matrix 
A in Prob. 9. Use the inverse power method. 

12 . BLet 



1.4 

0.8 

0.4' 


0.4 

-0.1 

0 .0" 

A = 

0.8 

6.6 

0.8 

B = 

-0.1 

0.4 

-0.1 


0.4 

0.8 

5.0 


0.0 

-0.1 

0.4 


Find the eigenvalues and eigenvectors of Ax = /.Lix by the Jacobi method. 

13. ■ Use the inverse power method to compute the smallest eigenvalue in Prob. 12. 

14. ■ Use the Jacobi method to compute the eigenvalues and eigenvectors of the 
matrix 


11 2 3 1 4 2 

2 9 3 5 2 1 

3 3 15 4 3 2 

1 5 4 12 4 3 

4 2 3 4 17 5 

2 1 2 3 5 8 
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15. ■ Find the eigenvalues of Ax = / Bx by the Jacobi method, where 


A = 


6 

-4 

1 

0 


-4 

6 

-4 

1 


1 

-4 

6 

-4 


0 

1 

-4 

7 


Warning: B is not positive definite. 

16. ■ 



-2 3 -1 
6-2 3 

-2 6 -2 
3-2 9 


1 2 


L 


n 


x 


The figure shows a cantilever beam with a superimposed finite difference mesh. ff 
u[x, t) is the lateral displacement of the beam, the differential equation of motion 
governing bending vibrations is 


where y is the mass per unit length and EI is the bending rigidity. The boundary 
conditionsare u{ 0, i) = d (0, 1) = ii'{L, t ) = d" ( L , t) = 0. With u[x, t) = y( x) sin o>t 
the problem becomes 


y 


( 4 ) = ^_Y_ 

EI ' 


y(0) = j/(0) = /'(L) = f'(L) = 0 


The corresponding finite difference equations are 


where 


7-4100 
-4 6-4 1 0 

1-4 6-4 1 

0 1-4 6 

0 0 1-4 

0 ■■■ 0 0 1 


0 ' 


" yi 


yi 

0 


yz 


yz 

0 


ys 


T3 




= x 


4 1 


y n ~ 2 


yn- 2 

5 -2 


y n ~ i 


y »-1 

2 1_ 


. y n . 


_ yn/2 _ 


2 

a) y 
X = —- 
EI 



4 


(a) Write down the matrix H of the Standard form Hz = Xz and the transformation 
matrix P as in y = Pz. (b) Write a program that computes the lowest two circular 
frequencies of the beam and the corresponding mode shapes (eigenvectors) using 
the Jacobi method. Run the program with n = 10. Note: the analytical solution for 
the lowest circular frequency is wi = (3.515/ L 2 ) jEI/y. 
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17. ■ 



L /2 L/4 P 



(a) 


LIA LIA 


01 2345678910 


(b) 


The simply supported column in Fig. (a) consists of three segments with the 
bending rigidities shown. If only the first buckling mode is of interest, it is sufficient 
to model half of the beam as shown in Fig. (b). The differential equation for the 
lateral displacement u{x) is 


with the boundary conditions u(0) = U{L/2) = 0. The corresponding finite dif- 
ference equations are 


2 

-1 

0 

0 

0 

0 

0 


0' 


Ui 


U\ 

-1 

2 

-1 

0 

0 

0 

0 


0 


u 2 


u 2 

0 

-1 

2 

-1 

0 

0 

0 


0 


«3 


u 3 

0 

0 

-1 

2 

-1 

0 

0 


0 


U 4 


II 4 

0 

0 

0 

-1 

2 

-1 

0 


0 


«5 

= X 

M5/1.5 

0 

0 

0 

0 

-1 

2 

-1 


0 


U& 


«6/2 

0 


0 

0 

0 

0 

-1 

2 

-1 


Ug 


Uq /2 

0 


0 

0 

0 

0 

0 

-1 

1 


_ U W _ 


Uio/A _ 


where 



Write a program that computes the lowest buckling load P of the column with 
the inverse power method. Utilize the banded forms of the matrices. 


18. ■ 



The springs supporting the three-bar linkage are undeformed when the linkage 
is horizontal. The equilibrium equations of the linkage in the presence of the 
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horizontal force P can be shown to be 



where fcis the spring stiffness. Determine the smallest buckling load P and the cor- 
responding mode shape. Hint: the equations can easily rewritten in the Standard 
form A9 = XO, where A is symmetric. 

19. ■ 



The differential equations of motion for the mass-spring system are 


k{—2ui + u 2 ) = miii 
k(ui — 2 u 2 + » 3 ) = 3 mu 2 
k{u 2 — 2u$) = 2 mu 3 

where u,(f) is the displacement of mass i from its equilibrium position and k 
is the spring stiffness. Determine the circular frequencies of vibration and the 
corresponding mode shapes. 

20. ■ 



Kirchoffs equations for the Circuit are 

T efi 1 1 . 2 . ., „ 

L ^p + c h+ c [ll ~ l2) ~° 

<fci 2 2 . . 3 . 

L—JW + 7; (*2 — 0) + 7;(l 2 — I3) = 0 
aP C C 

cfi 3 3 . 4 . 

L—yk + t;U3 — h) + — 4) = 0 

aP C C 

T (fi i 4 4 . . N 5 . 

l Hp + c [h ~ h)+ c k = 0 


Find the circular frequencies of the currents. 
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21 . ■ 



L 


Determine the circular frequencies of oscillation for the Circuit shown, given the 
Kirchoff equations 


L 

L 


d 2 i 2 

dt 2 

d 2 k 

dt 2 


dt 2 


d 2 i\ 

dt 2 

d 2 i 2 

dt 2 


+ L 


+ L 


+ L 


d 2 i\ 

d 2 i 2 \ 

l 1 . 

dt 2 

dt 2 ) 

> + c h 

d 2 i 2 

d 2 i 3 \ 


dt 2 

dt 2 ) 

> + c* 2 

d 2 i 2 

d 2 i 4 \ 


dt 2 

dt 2 ) 

> + c* 3 


d 2 U d 2 /3 
dt 2 dt 2 


d 2 4 4 

+ L ^fi + C h = ° 


22. ■ Several iterative methods exist for finding the eigenvalues of a matrix A. One of 
these is the LR method, which requires the matrix to be symmetric and positive 
definite. Its algorithm very simple: 


Let Ao = A 

do with i = 0,1, 2 ,... 

Use Choleski’s decomposition A; = L,Lf to compute L ( 
Form A,-+i = LfL,- 
end do 


It can be shown that the diagonal elements of A i+ i converge to the eigenvalues of 
A. Write a program that implements the LR method and test it with 


A = 


4 

3 

1 


3 

4 
2 


1 

2 

3 


9.4 Householder Reductiori to Tridiagonal Form 

It was mentioned before that similarity transformations can be used to transform an 
eigenvalue problem to a form that is easier to solve. The most desirable of the "easy” 
forms is, of course, the diagonal form that results from the Jacobi method. However, 
the Jacobi method requires about 10 n 3 to 20 n’ multiplications, so that the amount of 
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computation increases very rapidly with n. We are generally better off by reducing the 
matrix to the tridiagonal form, which can be done in precisely n — 2 transformations 
by the Householder method. Once the tridiagonal form is achieved, we stili have to 
extract the eigenvalues and the eigenvectors, but there are effective means of dealing 
with that, as we see the next article. 


Householder Matrix 

Each Householder transformation utilizes the Householder matrix 


Q = I — - 

v H 


(9.36) 


where u is a vector and 


H= -u J u= - lui' 


(9.37) 


Note that uu r in Eq. (9.36) is the outer product; that is, a matrix with the elements 
(uu r ) r = u,Uj . Since Q is obviously symmetric (Q r = Q), we canwrite 


mr 


H 


Q Q = QQ = i —tt 1 —tt = 1 - + 


tur 


H 


uu r u (u r u) U T 


H 


H 2 


uu r u (2 H) u T 

= 1-2 -+ - =1 

H H 2 


which shows that Q is also orthogonal. 

Now let x be an arbitrary vector and consider the transformation Qx. Choosing 


u = x + /ce i 


(9.38) 


where 


we get 


k = ± |x| 


|x| ei = [l 0 0 ••• o] 


Q x = ( 1 - ^ ) x = 


i- 


u (x+ te i) 3 


u (x r x+te, r x) 

= X-= X — 

H 


H 

u (fc 2 + kx i) 
H 


But 


2 H = (x+ te i) r (x+ te i) = |x| 2 + fc(x r ei+efx) + Z^efei 
= fc 2 + 2A:xi + fc 2 = 2 (fc 2 + kx i) 
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so that 


Qx = x — u = — fe| = I —k 0 



Hence the transformation eliminates ali elements of x except the first one. 


(9.39) 


Householder Reductiori of a Symmetric Matrix 


Let us now apply the following transformation to a symmetric n x «matrix A: 


1 

I— 1 

0 

_1 

A\\ 

1 

hn 

X 


1 

1_ 

1 

O 

O 

1_ 

x 

A' 


Qx QA' 


(9.40) 


Here x represents the first column of A with the first element omitted, and A' 
is simply A with its first row and column removed. The matrix Q of dimensions 
( n— 1) x (»— 1) is constructed using Eqs. (9.36)-(9.38). Referring to Eq. (9.39), we 
see that the transformation reduces the first column of A to 


Ali 

Qx 


A i 
-k 
0 


The transformation 


PxAPi 


An (Qx) r 

Qx QA'Q 


(9.41) 


thus tridiagonalizes the first row as well as the first column of A. Here is a diagram of 
the transformation for a 4 x 4 matrix: 


1 

0 0 0 

0 

0 

0 

Q 


^11 

A 12 A 13 Au 

A 21 

Au 

A 41 

A' 


1 

0 0 0 

OOO 

Q 


^11 

—A: 0 0 

-k 

0 

0 

QA'Q 


The second row and column of A are reduced next by applying the transformation to 
the 3x3 lower right portion of the matrix. This transformation can be expressed as 
A <- P 2 AP 2 , where now 
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P 2 = 


12 U 

0 Q 


(9.42) 


In Eq. (9.42) I 2 is a 2 x 2 identity matrix and Q is a {n — 2) x [n — 2) matrix constructed 
by choosing for x the bottom n — 2 elements of the second column of A. It takes a total 
of n — 2 transformation with 


P,= 


h 0 T 
0 Q 


, i = 1, 2 ,..., n— 2 


to attain the tridiagonal form. 

It is wasteful to form P, and then carry out the matrix multiplication P,AP,. We 
note that 


uu 1 


H 


Au 


A'Q = A' I-= A'-= A-vu 


H 


where 


Therefore, 


v = 


Au 

~H 


(9.43) 


uu 1 


= A'-vu r - 


uu J 


QAQ = (I - — ) (A—vu r ) = A'-vu r - u (A'-vu r ) 


H 


u(u r A') u(u r v)u T 


+ 


H H 

= A'-vu r -uv r + 2guu r 


where 


Letting 


g- 


2 H 


w = V — gu 

it can be easily verified that the transformation can be written as 

QAQ = A-wu T -uw T 


(9.44) 


(9.45) 


(9.46) 


which gives us the following computational procedure which is to be carried out with 
i = I, 2 ,..., n — 2: 


1. Let A' be the ( n— i ) x (n — i ) lower right-hand portion of A. 

2 . Letx = A i+2 ,i ••• A, u j (the column oflength n - ijusttotheleftofA'). 

3. Compute |x|. Let k = |x| iix\ >0 and A: = — |x| if x\ <0 (this choice of sign mini- 
mizes the roundoff error). 
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4. Let u = [ic+x i x 2 x 3 • • • x„_,l . 

5. Compute H = |u| 2 . 

6 . Compute v = A'u/ H. 

7. Compute g = u T v/(2H). 

8. Compute w = v — gu. 

9. Compute the transformation A' <- A'—w r u - u 7 w. 
10. Set A/; 11 = Ai + u — k. 


Accumulated Transformation Matrix 

Since we used similarity transformations, the eigenvalues of the tridiagonal matrix 
are the same as those of the original matrix. However, to determine the eigenvectors 
X of original A we must use the transformation 

X = PXtndiag 

where P is the accumulation of the individual transformations: 


P = P!P 2 - ■ Pti—2 


We build up the accumulated transformation matrix by initializing P to a nx n iden- 
tity matrix and then applying the transformation 


Pll P12 


X 0 T ~ 


Pll P21Q 

P21 P22 


1 

0 

0 

1 _ 


P12 P22Q 


(b) 


with i = 1,2,..., n — 2. It can be seen that each multiplication affects only the right- 
most n— i columns of P (since the first rowof Pi 2 contains onlyzeroes, it can also be 
omitted in the multiplication). Using the notation 


P'= 


Pl 2 

?22 


we have 


P12Q 

P22Q 


P'Q = P' 



, Pu T 

= P'-u r 

H 


P'-yu r 


where 


y = 


Pu 

JT 


The procedure for carrying out the matrix multiplication in Eq. (b) is: 


(9.47) 


(9.48) 


Retrieve u (in our triangularization procedure the u’s are stored in the columns 
of the lower triangular portion of A). 
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• Compute H = |u| 2 

• Compute y = P'u /H. 

• Compute the transformation PV- P — yu r . 

■ householder 

The function householder in this module does the triangulization. It returns (d, c), 
where d and c are vectors that contain the elements of the principal diagonal and 
the subdiagonal, respectively. Only the upper triangular portion is reduced to the 
triangular form. The part below the principal diagonal is used to store the vectors u. 
This is done automatically by the statement u = a [ k+1: n, k ] which does not create 
a new object u, but simply sets up a reference to a[k+l: n, k] (makes a deep copy). 
Thus any changes made to u are reflected in a [ k+l: n, k ]. 

The function computeP returns the accumulated transformation matrixP. There 
is no need to call it if only the eigenvalues are to be computed. 

## module householder 
’’’ d,c = householder(a). 

Householder similarity transformation of matrix [a] to 
the tridiagonal form [c\d\c]. 

p = computeP(a). 

Computes the accumulated transformation matrix [p] 
after calling householder(a). 

from numarray import dot,matrixmultiply,diagonal, \ 
outerproduct,identity 
from math import sqrt 

def householder(a) : 
n = len(a) 

for k in range(n-2): 
u = a[k+l:n,k] 
uMag = sqrt(dot(u,u)) 
if u[0] < 0.0: uMag = -uMag 
u[0] =u[0] + uMag 
h = dot(u,u)/2.0 

v = matrixmultiply(a[k+l:n,k+l:n],u)/h 
g = dot(u,v)/(2.0*h) 
v = v - g*u 

a[k+l:n,k+l:n] = a[k+l:n,k+l:n] - outerproduct(v,u) \ 

- outerproduct(u,v) 
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a[k,k+l] = -uMag 

return diagonal(a),diagonal(a,1) 


def computeP(a): 
n = len(a) 
p = identity(n)*1.0 
for k in range(n-2): 
u = a[k+l:n,k] 
h = dot(u,u)/2.0 

v = matrixmultiply(p[1:n,k+1:n],u)/h 
p[l:n,k+l:n] = p[l:n,k+l:n] - outerproduct(v,u) 
return p 


EXAMPLE 9.7 

Transform the matrix 

' 7 2 3 -1 

A 2 8 5 1 

A = 

3 5 12 9 

-1 1 9 7 

into tridiagonal form using Householder reduction. 


Solution Reduce the first row and column: 



"8 

5 

l" 


2 ~ 

A' = 

5 

12 

9 

X = 

3 


1 

9 

7 


-1 


k = |x| = 3.7417 


k+ x i 


'5.7417" 

x 2 

= 

3 

x 3 


-1 


H = - |ur = 21.484 
2 


uu 


T 


32.967 17 225 -5.7417 

17.225 9 -3 

-5.7417 -3 1 


Q 



-0.53450 -0.80176 0.26725 

-0.80176 0.58108 0.13964 

0.26725 0.13964 0.95345 


QA'Q = 


10.642 

-0.1388 

-9.1294 


-0.1388 

5.9087 

4.8429 


-9.1294 

4.8429 

10.4480 
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7 

-3.7417 
0 
0 

In the last step we used the formula Qx = 
Reduce the second row and column: 


A n (Qx) r 

Qx QA'Q 


-3.7417 

0 

0 

10.642 

-0.1388 

-9.1294 

-0.1388 

5.9087 

4.8429 

-9.1294 

4.8429 

l T 

10.4480 

-k 0 

' °1 ■ 



A = 

5.9087 

4.8429 

X = 

’-0.1388" 


4.8429 

10.4480 


-9.1294 


k= — |x| = -9.1305 


where the negative sign on k was determined by the sign of jti. 


k+ x 1 


-9.2693’ 

X 2 


-9.1294 


H=- |u| z = 84.633 
2 


uu 


T 


85.920 84.623 
84.623 83.346 


Q 


uu r _ 0.01521 -0.99988 

~TT ~ -0.99988 0.01521 


QA'Q 


10.594 4.772 
4.772 5.762 


A 


An 

A21 


0 


A12 

A22 

Qx 


0 r 

(Qx) r 

QAQ 


7 

-3.742 

0 

0 

3.742 

10.642 

9.131 

0 

0 

9.131 

10.594 

4.772 

0 

-0 

4.772 

5.762 


EXAMPLE 9.8 

Use the function householder to tridiagonalize the matrix in Example 9.7; also de- 
termine the transformation matrix P. 


Solution 

#!/usr/bin/python 
## example9_8 

from numarray import array,matrixmultiply 
from householder3 import * 


a = array([[ 7.0, 2.0, 3.0, -1.0], \ 

[2.0, 8.0, 5.0, 1.0] , \ 
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[ 3.0, 5.0, 12.0, 9.0], \ 
[-1.0, 1.0, 9.0, 7.0]]) 
d,c = householder(a) 

print ’'Principal diagonal {d}:\n’’,d 
print ’’ \nSubdiagonal {c}:\n’’,c 
print ’’ \nTransformation matrix [P]:’' 
print computeP(a) 

raw_input(’’ \nPress return to exit’’) 


The results of running the above program are: 


Principal diagonal {d}: 

[ 7. 10.64285714 10.59421525 5.76292761] 


Subdiagonal {c}: 


[-3.74165739 

9.13085149 

Transformation 

matrix [P] 

[[ 1. 

0 . 

[ 0 . 

-0.53452248 

[ 0 . 

-0.80178373 

[ 0 . 

0.26726124 


4.77158058] 


0 . ] 
0.80574554] 
0.57888514] 
0.12516436]] 


0. 

-0.25506831 
-0.14844139 - 
-0.95546079 - 


9.5 Eigenvalues of Symmetric Tridiagonal Matrices 

Sturm Sequence 

In principle, the eigenvalues of a matrix A can be determined by finding the roots of 
the characteristic equation | A — 1I| = 0. This method is impractical for large matrices, 
since the evaluation of the determinant involves ri '/3 multiplications. However, if the 
matrix is tridiagonal (we also assume it to be symmetric), its characteristic polynomial 


PnW = |A-XI| 


d\ — k Ci 0 0 

Ci da — k C 2 0 

0 C 2 da, ~ k C 3 

0 0 C 3 di — k 


0 

0 

0 

0 


0 


0 


0 


Cn— 1 d,i k 
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can be computed with only 3(n— 1) multiplications using the following sequence of 
operations: 


PoM = 1 
P] (A) = d\ — A 

PIX) = [di - X)Pi_ i(A) - i = 2,3,..., n 


(9.49) 


The polynomials Po(X). P\ (A),.... P„[X) form a Sturm sequence that has the fol¬ 
lowing property: 

• The number of sign changes in the sequence P 0 ( a ), P t (a), P n (a) is equal to the 
number of roots of P n [X) that are smaller than a. If a member Pi{d) of the sequence 
is zero, its sign is to be taken opposite to that of P,_ \{ci). 

As we see later, Sturm sequence property makes it possible to bracket the eigen- 
values of a tridiagonal matrix. 

■ sturmSeq 

Given d, c and A, the function sturmSeq returns the Sturm sequence 

P 0 {X), P, (A),..., P n {X) 

The function numLambdas returns the number of sign changes in the sequence (as 
noted before, this equals the number of eigenvalues that are smaller than X). 

## module sturmSeq 

’’’ p = sturmSeq(c,d,lam). 

Returns the Sturm sequence {p[0],p[l],...,p[n]} 
associated with the characteristic polynomial 
I[A] - lam[I]I =0, where [A] = [c\d\c] is a n x n 
tridiagonal matrix. 

numLam = numLambdas(p). 

Returns the number of eigenvalues of a tridiagonal 
matrix [A] = [c\d\c] that are smaller than ’lam’. 

Uses the Sturm sequence {p} obtained from 'sturmSeq’. 

from numarray import ones, Float64 


def sturmSeq(d,c,lam): 
n = len(d) + 1 
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p = ones((n),type=Float64) 
p[l] = d[0] - lam 
for i in range(2,n): 

p[i] = (d[i-l] - lam)*p[i-l] - (c[i-2]**2)*p[i-2] 
return p 

def numLambdas(p): 
n = len(p) 
signOld = 1 
numLam = 0 
for i in range(l,n): 

if p[i] > 0.0: sign = 1 
elif p[i] < 0.0: sign = -1 
else: sign = -signOld 

if sign*sign01d < 0: numLam = numLam + 1 
signOld = sign 
return numLam 

EXAMPLE 9.9 

Use the Sturm sequence property to show that the smallest eigenvalue of A is in the 
interval (0.25, 0.5), where 


2-100 
-1 2-1 0 
0-1 2-1 
0 0-12 


A = 


Solution Taking X = 0.5, we have d* — X = 1.5 and cf_ 1 = 1 and the Sturm sequence 
in Eqs. (9.49) becomes 


Po(0.5) = 1 
Pi(0.5) = 1.5 

P 2 (0.5) = 1.5(1.5) - 1 = 1.25 
P 3 (0.5) = 1.5(1.25) - 1.5 = 0.375 
P 4 (0.5) = 1.5(0.375) - 1.25 = -0.6875 


Since the sequence contains one sign change, there exists one eigenvalue smaller 
than 0.5. 
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Repeating the process with X = 0.25, we get di — X = 1.75 and c?_, = 1, which 
results in the Sturm sequence 

id) (0.25) = 1 

i>i(0.25) = 1.75 

P 2 (0.25) = 1.75(1.75) - 1 = 2.0625 
P 3 (0.25) = 1.75(2.0625) - 1.75 = 1.8594 
P 4 (0.25) = 1.75(1.8594) - 2.0625 = 1.1915 

There are no sign changes in the sequence, so that ali the eigenvalues are greater than 
0.25. We thus conclude that 0.25 < li < 0.5. 

Gerschgorin's Theorem 

Gerschgorin’s theorem is useful in determining the global bounds on the eigenval¬ 
ues of an n x n matrix A. The term “global” means the bounds that enclose all the 
eigenvalues. We give here a simplified version for a symmetric matrix. 

• If A is an eigenvalue of A, then 

d — Ti < X < at + q, i = 1,2,..., n 

where 

n 

d = Au n = ^ \Aij\ (9.50) 

J'=i 

It follows that the limits on the smallest and the largest eigenvalues are given by 
Vin > min(n/ - n) Vax < max(o,- + n) (9.51) 


■ gerschgorin 

The function gerschgorin returns the lower and upper global bounds on the eigen¬ 
values of a symmetric tridiagonal matrix A = [c\d\c]. 

## module gerschgorin 

’’’ lamMin,lamMax = gerschgorin(d,c). 

Applies Gerschgorin’s theorem to find the global bounds on 
the eigenvalues of a tridiagomal matrix [A] = [c\d\c]. 


def gerschgorin(d,c): 
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the eigenvalues by Gerschgorin’s theorem. Then the method of bisection in conjunc- 
tion with the Sturm sequence property is used to determine r N , r N ~ i,..., r 0 in that 
order. 

## module lamRange 

’’’ r = lamRange(d,c,N). 

Returns the sequence {r[0],r[l]....,r[N]} that 
separates the N lowest eigenvalues of the tridiagonal 
matrix [A] = [c\d\c]; that is, r[i] < lam[i] < r[i+l]. 

from numarray import ones,Float64 
from sturmSeq import * 
from gerschgorin import * 

def lamRange(d,c,N): 

lamMin,lamMax = gerschgorin(d,c) 
r = ones((N+l),type=Float64) 
r[0] = lamMin 

# Search for eigenvalues in descending order 
for k in range(N,0,-1): 

# First bisection of interval(lamMin,lamMax) 
lam = (lamMax + lamMin)/2.0 

h = (lamMax - lamMin)/2.0 
for i in range(lOOO): 

# Find number of eigenvalues less than lam 
p = sturmSeq(d,c,lam) 

numLam = numLambdas(p) 

# Bisect again & find the half containing lam 
h = h/2.0 

if numLam < k: lam = lam + h 
elif numLam > k: lam = lam - h 
else: break 

# If eigenvalue located, change the upper limit 

# of search and record it in [r] 
lamMax = lam 

r[k] = lam 
return r 

EXAMPLE 9.11 

Bracket each eigenvalue of the matrix A in Example 9.10. 
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n = len(d) 

lamMin = d[0] - abs(c[0]) 
lamMax = d[0] + abs(c[0]) 
for i in range(1,n-1) : 

lam = d[i] - abs(c[i]) - abs(c[i-l]) 
if lam < lamMin: lamMin = lam 
lam = d[i] + abs(c[i]) + abs(c[i-l]) 
if lam > lamMax: lamMax = lam 
lam = d[n-l] - abs(c[n-2]) 
if lam < lamMin: lamMin = lam 
lam = d[n-l] + abs(c[n-2]) 
if lam > lamMax: lamMax = lam 
return lamMin,lamMax 


EXAMPLE 9.10 

Use Gerschgorin’s theorem to determine the bounds on the eigenvalues of the matrix 


A = 


4 

-2 

0 


-2 

4 

-2 


0 

-2 

5 


Solution Referring to Eqs. (9.50), we get 


ai = 4 a 2 = 4 r% = 5 

n = 2 r 2 = 4 r 3 = 2 


Hence 


k min > minia,- - n) = 4 - 4 = 0 
Vax < max(a ; + r{) = 4 + 4 = 8 


Bracketing Eigenvalues 

The Sturm sequence property together with Gerschgorin’s theorem provides us con¬ 
venient tools for bracketing each eigenvalue of a symmetric tridiagonal matrix. 


■ lamRange 

The function lamRange brackets the N smallest eigenvalues of a symmetric tridi¬ 
agonal matrix A = [c\d\c]. It returns the sequence ro, r\ .r ; y, where each interval 

(r,_i, ri) contains exactly one eigenvalue. The algorithm first finds the bounds on all 
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Solution InExample9.10 wefoundthat allthe eigenvalues lie in (0, 8). Wenowbisect 
this interval and use the Sturm sequence to determine the number of eigenvalues in 
(0, 4). With k = 4, the sequence is—see Eqs. (9.49) 

Po(4) = 1 

Pi( 4) =4-4 = 0 

P 2 (4) = (4 — 4) (0) — 2 2 (1) = —4 

P 3 ( 4) = (5 — 4) (—4) — 2 2 (0) = —4 

Since a zero value is assigned the sign opposite to that of the preceding member, the 
signs in this sequence are The one sign change shows the presence of 

one eigenvalue in (0, 4). 

Next we bisect the interval (4, 8) and compute the Sturm sequence with k = 6: 
fl>(6) = 1 

Pi (6) = 4 — 6 = —2 

P 2 (6) = (4 — 6) (—2) — 2 2 (1) = 0 

P 3 (6) = (5 — 6)(0) — 2 2 (—2) = 8 

In this sequence the signs are (+,—,+,+), indicating two eigenvalues in (0, 6). 
Therefore 

0 < li < 4 4 < k 2 < 6 6 < 1 3 < 9 


Computation of Eigenvalues 

Once the desired eigenvalues are bracketed, they can be found by determining the 
roots of P n (k) = 0 with bisection or Brent’s method. 

■ eigenvals3 

The function eigenval s 3 computes the N smallest eigenvalues of a symmetric tridi¬ 
agonal matrix with the method of Brent. 

## module eigenvals3 

’’’ lam = eigenvals3(d,c,N). 

Returns the N smallest eigenvalues of a 
tridiagonal matrix [A] = [c\d\c]. 


from lamRange import 
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from brent import * 

from sturmSeq import sturmSeq 

from numarray import zeros,Float64 

def eigenvals3(d,c,N): 

def f(x): # f(x) = I[A] - x[I]I 

p = sturmSeq(d,c,x) 
return p[len(p)-l] 

lam = zeros((N),type=Float64) 
r = lamRange(d,c,N) # Bracket eigenvalues 
for i in range(N): # Solve by Brent’s method 

lam[i] = brent(f,r[i],r[i+1]) 
return lam 

EXAMPLE 9.12 

Use eigenvals3 to determine the three smallest eigenvalues of the 100 x 100 matrix 

"2-1 0 -0" 

-1 2 -1 0 

A= 0-1 2 ••• 0 

0 0 ••• -1 2_ 

Solution 

#!/usr/bin/python 
## example9_12 

from numarray import ones,Float64 
from eigenvals3 import * 

N = 3 
n = 100 

d = ones((n))*2.0 
c = ones((n-1))*(-1.0) 
lambdas = eigenvals3(d,c,N) 
print lambdas 

raw_input('’ \nPress return to exit’’) 

Here are the eigenvalues: 

[ 0.00096744 0.00386881 0.0087013 ] 
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Computation of Eigenvectors 

If the eigenvalues are known (approximate values will be good enough), the best 
means of computing the corresponding eigenvectors is the inverse power method 
with eigenvalue shifting. This method was discussed before, but the algorithm listed 
did not take advantage of banding. ffere we present a version of the method written 
for symmetric tridiagonal matrices. 

M inversePower3 

This function is very similar to inversePower listed in Art. 9.3, but it executes much 
faster since it exploits the tridiagonal structure of the matrix. 

## module inversePower3 

’’’ lam,x = inversePower3(d,c,s,tol=l.Oe-6). 

Inverse power method applied to a tridiagonal matrix 
[A] = [c\d\c]. Returns the eigenvalue closest to ’s’ 
and the corresponding eigenvector. 

from numarray import dot,zeros,Float64 
from LUdecomp3 import * 
from math import sqrt 
from random import random 

def inversePower3(d,c,s,tol=l.Oe-6) : 
n = len(d) 
e = c.copyO 

cc = c.copyO # Save original [c] 


dStar = d - s 
LUdecomp3(cc,dStar , e) 
x = zeros((n),type=Float64) 
for i in range(n): 


# Form [A*] = [A] - s[I] 

# Decompose [A*] 


# Seed [x] with random numbers 


x[i] = randomO 
xMag = sqrt(dot(x,x)) 
x =x/xMag 
flag = 0 

for i in range(30): 


# Normalize [x] 


# Begin iterations 

# Save current [x] 


xOld = x.copyO 


LUsolve3(cc,dStar,e,x) # Solve [A*][x] = [xOld] 
xMag = sqrt(dot(x,x)) # Normalize [x] 

x = x/xMag 
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if dot(x01d,x) < 0.0: # Detect change in sign of [x] 

sign = -1.0 
x = -x 

else: sign = 1.0 

if sqrt(dot(xOld - x,x01d - x)) < tol: 
return s + sign/xMag,x 

print 'Inverse power method did not converge’ 

EXAMPLE 9.13 

Compute the lOth smallest eigenvalue of the matrix A given in Example 9.12. 

Sol ution The following program extracts the Mh eigenvalue of A by the inverse power 
method with eigenvalue shifting: 

#!/usr/bin/python 
## example9_13 
from numarray import ones 
from lamRange import * 
from inversePower3 import * 

N = 10 
n = 100 

d = ones((n))*2.0 
c = ones((n-1))*(-1.0) 
r = lamRange(d,c,N) 
s = (r[N-l] + r[N])/2.0 
lam,x = inversePower3(d,c,s) 
print ''Eigenvalue No.’’,N,’’ =’’,lam 
raw_input('’ \nPress return to exit’’) 

The resuit is 

Eigenvalue No. 10 = 0.0959737849345 

EXAMPLE 9.14 

Compute the three smallest eigenvalues and the corresponding eigenvectors of the 
matrix A in Example 9.5. 

Solution 

#!/usr/bin/python 
## example9_14 


# Bracket N smallest eigenvalues 

# Shift to midpoint of Nth bracket 

# Inverse power method 




375 


9.5 Eigenvalues of Symmetric Tridiagonal Matrices 


from householder3 import * 
from eigenvals3 import * 
from inversePower3 import * 

from numarray import array,zeros,Float64,matrixmultiply 


N = 3 # Number of eigenvalues requested 


arrayC[[ 

11.0, 

2.0, 

3.0, 

1.0, 

4.0] , 

\ 

[ 

2.0, 

9.0, 

3.0, 

5.0, 

2.0] , 

\ 

[ 

3.0, 

3.0, 

15.0, 

4.0, 

3.0] , 

\ 

[ 

1.0, 

5.0, 

4.0, 

12.0, 

4.0] , 

\ 

[ 

4.0, 

2.0, 

3.0, 

4.0, 

17.0]]) 



xx = zeros((len(a),N),type=Float64) 


d,c = householder(a) 
p = computeP(a) 
lambdas = eigenvals3(d,c,N) 
for i in range(N): 

s = lambdas[i]*1.0000001 
lam,x = inversePower3(d,c,s) 
xx[:,i] = x 

xx = matrixmultiplyCp,xx) 
print '’Eigenvalues:\n'’.lambdas 
print '’ \nEigenvectors :\n’',xx 
raw_input(’’Press return to exit 


# Tridiagonalize [A] 

# Compute transformation matrix 

# Compute eigenvalues 

# Shift very close to eigenvalue 

# Compute eigenvector [x] 

# Place [x] in array [xx] 

# Recover eigenvectors of [A] 

’) 


Eigenvalues : 

[ 4.87394638 

Eigenvectors : 
[[ 0.26726603 
[-0.74142854 
[-0.05017271 
[ 0.59491453 
[-0.14970633 


8.66356791 


0.72910002 
0.41391448 
-0.4298639 
0.06955611 
-0.32782151 


10.93677451] 


0.50579164] 

-0.31882387] 

0.52077788] 

-0.60290543] 

-0.08843985]] 


PROBLEM SET9.2 

1. Use Gerschgorin’s theorem to determine bounds on the eigenvalues of 


'10 4 -l' 


1 

to 

1 

to 

4 2 3 

(b) B = 

2 5 3 

-13 6 


1 

cn 

CN 

1 

_1 


(a) A = 
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2. Use the Sturm sequence to show that 


A = 


5 

-2 

0 

0 


-2 

4 

-1 

0 


0 0 
-1 0 
4 -2 
-2 5 


has one eigenvalue in the interval (2, 4). 

3. Bracket each eigenvalue of 


A = 


4 

-1 

0 


-1 

4 

-1 


0 

-1 

4 


4. Bracket each eigenvalue of 


A = 


6 1 
1 8 
0 2 


0 

2 

9 


5. Bracket every eigenvalue of 


2-100 
-1 2-1 0 
0-1 2-1 
0 0-11 


6. Tridiagonalize the matrix 


A = 


12 

4 

3 


4 

9 

3 


3 

3 

15 


with Householder’s reduction. 

7. Use Householder’s reduction to transform the matrix 


4 

-2 

1 

-1 

-2 

4 

-2 

1 

1 

-2 

4 

-2 

-1 

1 

-2 

4 


to tridiagonal form. 
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8. ■ Compute all the eigenvalues of 


A = 


'6 2 0 
2 5 2 
0 2 7 
0 0 4 
0 0 0 


0 0 ' 
0 0 
4 0 
6 1 
1 3_ 


9. ■ Find the smallest two eigenvalues of 

'4-1 0 1' 

-1 6-20 

A = 

0-2 3 2 

10 2 4 


10. ■ Compute the three smallest eigenvalues of 


7 

-4 

3 

-2 

1 

0 

-4 

8 

-4 

3 

-2 

1 

3 

-4 

9 

-4 

3 

-2 

-2 

3 

-4 

10 

-4 

3 

1 

-2 

3 

-4 

11 

-4 

0 

1 

-2 

3 

-4 

12 


and the corresponding eigenvectors. 

11. ■ Find the two smallest eigenvalues of the 6x6 Hilbert matrix 



' 1 

1/2 

1/3 ••• 

1/6" 


1/2 

1/3 

1/4 ■■■ 

1/7 

A = 

1/3 

1/4 

1/5 

1/8 


.1/6 

1/7 

1/8 ■■■ 

1/11. 


Recall that this matrix is ill-conditioned. 

12. ■ Rewrite the function lamRange (d, c, N) so that it will bracket the N largest 
eigenvalues of a tridiagonal matrix. Use this function to bracket the two largest 
eigenvalues of the Hilbert matrix in Example 9.11. 
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The differential equations of motion of the mass-spring system are 

k{—2ui + u 2 ) = mui 
k{u\ — 2 u 2 + u$) = 3mu 2 
k{u 2 — 2 u$) = 2 mib, 


where Ui ( t) is the displacement of mass i from its equilibrium position and A; is 
the spring stiffness. Substituting u t ( t) = y,- sin wt, we obtain the matrix eigenvalue 
problem 


2-1 0 

-1 2 -1 
0-1 2 


yi 



meo 

T2 

lc 




"l 0 o‘ 


yi 

0 3 0 


y2 

O 

O 

ro 

i_ 




Determine the circular frequencies co and the corresponding relative amplitudes 
y ; : of vibration. 



The figure shows «identical masses connected by springs of different stiffnesses. 
The equation governing free vibration of the system is Au = rrto 2 u, where <» is the 
circular frequency and 


k\ + k 2 —k 2 


— k 2 k 2 + k 2 
0 -fe 


0 

~h, 

k 2 + ki 


0 

0 

-h 


o 

0 

0 


0 

0 


0 k n — i k n —i + k n k„ 

0 0 -k„ k n 


Given the spring stiffnesses k = k 2 ■■ ■ k n J , write a program that com¬ 
putes the N lowest eigenvalues X = meo 2 and the corresponding eigenvectors. 
Run the program with AT = 4 and 


k=[400 400 400 0.2 400 400 20o] ^ kN/m 
Note that the system is weakly coupled, /c, being small. Do the results make sense? 


1 2 


L 


□—x 
n 


15. ■ 
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The differential equation of motion of the axially vibrating bar is 



where u[x, l) is the axial displacement, p represents the mass density and E is the 
modulus of elasticity. The boundary conditions are u{(), L) = i/{L, L) = 0. Letting 
u(x, t) = y(x) sin cot, we obtain 

f = -ca 2 ^y y(0) = /(L) = 0 


The corresponding finite difference equations are 


2 

-1 

0 

0 


0' 


yt " 


yi 

-1 

2 

-1 

0 


0 


yz 


yz 

0 

-1 

2 

-1 


0 


ys 

/ a>L\ 2 p 

yz 

0 

0 


-1 

2 

-1 


yn -1 

V n) E 

y n ~ 1 

0 

0 


0 

-1 

1 


Y n 


_ yn /2 _ 


(a) If the Standard form of these equations is Hz = /,z, write down H and the 
transformation matrix P in y = Pz. (b) Compute the lowest circular frequency of 
the bar with n = 10,100 and 1000 utilizing the module inversePower3. Note : the 
analytical solution is a>\ = n^/E/p/ (2 L). 

16. ■ 



The simply supported column is resting on an elastic foundation of stiffness k 
(N/m per meter length). An axial force P acts on the column. The differential 
equation and the boundary conditions for the lateral displacement u are 


u A ‘ + —— U' H- u= 0 


EI 


EI 


u{ 0 ) = U'{ 0 ) = u{E) = iI'(L) = 0 

Using the mesh shown, the finite difference approximation of these equations is 

(5 + a)Ui — 4 U-2 + 113 = X[2ui — u 2 ) 

— 4 mi + (6 + a)u 2 — 4 113 + 114 = X(—Ui + 2 u 2 + Ua) 
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U\ — 4 u 2 + (6 + a)u 3 — 4 114 + 115 = X[— u 2 + 2u$ — 14 ) 


Un- 3 - 4 Un —2 + (6 + a) i - 4 u n = X(-u n _ 2 + 2u n -i - U n ) 
u n - 2 - 4+ (5 + a)r<„ = A.(-«n-i + 2u n ) 


where 

_ fc/# _ 1 kL A i _ P/ 2 2 _ 1 PL 2 

01 ~ ~EI ~ (n + 1 ) 4 R/" “ UT ~ (n+ l ) 2 ”eT 

Write a program that computes the lowest three buckling loads P and the corre- 
sponding mode shapes. Run the program with kL' 1 /{HI) = 1000 and n = 25. 

17. ■ Find smallest five eigenvalues of the 20 x 20 matrix 


'2 10 0 ■ • • 0 1 ' 

121 0 ••• 0 0 

012 1 ■■■ 0 0 

A = : • | 

0 0 ■■■ 1 2 10 

0 0 0 1 21 

_1 0 0 0 1 2 _ 

Note: this is a difficult matrix that has many pairs of double eigenvalues. 


9.6 Other Methods 

On occasions when ali the eigenvalues and eigenvectors of a matrix are required, the 
QR algorithm is a worthy contender. It is based on the decomposition A = QR where Q 
and R are orthogonal and upper triangular matrices, respectively. The decomposition 
is carried out in conjuction with Householder transformation. There is also a QL 
algorithm : A = QL that works in the same manner, but here L is a lower triangular 
matrix. 

Schur’s factorization is another solid technique for determining the eigenvalues 
of A. Here the decomposition is A = Q 7 tJQ, where Q is orthogonal and U is an upper 
triangular matrix. The diagonal terms of U are the eigenvalues of A. 

The LR algorithm is probably the fastest means of computing the eigenvalues; it 
is also very simple to implement—see Prob. 22 of Problem Set 9.1. But its stability is 
inferior to the other methods. 








10 Introduction to Optimization 


Find x that minimizes F (x) subject to g(x) = 0, /z(x) > 0 


10.1 Introduction 

Optimization is the term often used for minimizing or maximizing a function. It is suf¬ 
ficient to consider the problem of minimization only; maximization of F (x) is achieved 
by simply minimizing — F (x). In engineering, optimization is closely related to design. 
The function F (x), called the merit function or objective function, is the quantity that 
we wish to keep as small as possible, such as cost or weight. The components of x, 
known as the design variables, are the quantities that we are free to adjust. Physical 
dimensions (lengths, areas, angles, etc.) are common examples of design variables. 

Optimization is a large topic with many books dedicated to it. The best we can do in 
limited space is to introduce a few basic methods that are good enough for problems 
that are reasonably well behaved and don’t involve too many design variables. By 
omitting the more sophisticated methods, we may actually not miss ali that much. 
Ali optimization algorithms are unreliable to a degree—any one of them may work on 
one problem and fail on another. As a rule of thumb, by going up in sophistication we 
gain computational efficiency, but not necessarily reliability. 

The algorithms for minimization are iterative procedures that require starting 
values of the design variables x. If F (x) has several local minima, the initial choice of 
x determines which of these will be computed. There is no guaranteed way of finding 
the global optimal point. One suggested procedure is to make several computer runs 
using different starting points and pick the best resuit. 

More often than not, the design variables are also subjected to restrictions, or 
constraints, which may have the form of equalities or inequalities. As an example, 
take the minimum weight design of a roof truss that has to carry a certain loading. 
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Assume that the layout of the members is given, so that the design variables are the 
cross-sectional areas of the members. Here the design is dominated by inequality 
constraints that consist of prescribed upper limits on the stresses and possibly the 
displacements. 

The majority of available methods are designed for unconstrained optimization, 
where no restrictions are placed on the design variables. In these problems the min¬ 
ima, if they exist, are stationary points (points where gradient vector of F (x) vanishes). 
In the more difficult problem of constrained optimization the minima are usually lo- 
cated where the F (x) surface meets the constraints. There are special algorithms for 
constrained optimization, but they are not easily accessible due to their complexity 
and specialization. One way to tackle a problem with constraints is to use an uncon¬ 
strained optimization algorithm, but modify the merit function so that any violation 
of constraints is heavily penalized. 

Consider the problem of minimizing F (x) where the design variables are subject 
to the constraints 


g, : (x) = 0 , i=l,2, ...,M (10.Ia) 

hj(x) < 0, 7 = 1,2, ...,N (10.lb) 

We choose the new merit function be 

F* (x) = F(x) + XP[x) (10.2a) 


where 


M N 

P(x) = ^2 [g/W] 2 + J2 { max [°’ fyM]} 2 (10.2b) 

i=i 7=1 

is the penalty function and X is a multiplier. The function max(a, b) returns the larger of 
a and b. It is evident that P(x) = 0 if no constraints are violated. Violation of a constraint 
imposes a penalty proportional to the square of the violation. Hence the minimization 
algorithm tends to avoid the violations, the degree of avoidance being dependent on 
the magnitude of X. If X is small, optimization will proceed faster because there is more 
“space” in which the procedure can operate, but there may be significant violation of 
constraints. On the other hand, a large X can resuit in a poorly conditioned procedure, 
but the constraints will be tightly enforced. It is advisable to run the optimization 
program with X that is on the small side. If the results show unacceptable constraint 
violation, increase X and run the program again, starting with the results of the pre- 
vious run. 

An optimization procedure may also become ill-conditioned when the con¬ 
straints have widely different magnitudes. This problem can be alleviated by scaling 
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the offending constraints; that is, multiplying the constraint equations by suitable 
constants. 


10.2 Minimization Along a Line 


f(x) 



X 


Figure 10.1. Example oflocal and global minima. 


Consider the problem of minimizing a function /(x) of a single variable x with the 
constraints c < x < d. A hypothetical plot of the function is shown in Fig. 10.1. There 
are two minimum points: a stationary point characterized by f' (x) = 0 that represents 
a local minimum, and a global minimum at the constraint boundary. It appears that 
finding the global minimum is simple. All the stationary points could be located by 
finding the roots of df/dx = 0 , and each constraint boundary may be checked for a 
global minimum by evaluating /(c) and f{d). Then why do we need an optimization 
algorithm? We need it if /(x) is difficult or impossible to differentiate; for example, if 
/ represents a complex computer algorithm. 

Bracketing 

Before a minimization algorithm can be entered, the minimum point must be brack- 
eted. The procedure of bracketing is simple: start with an initial value of Xq and move 
downhill computing the function at xi, Xz, X 3 ,... until we reach the point x n where 
/(x) increases for the first time. The minimum point is now bracketed in the inter- 
val (x„_ 2 , x n ). What should the step size hi = x !+ i — x,- be? It is not a good idea have 
a constant hi since it often results in too many steps. A more efficient scheme is to 
increase the size with every step, the goal being to reach the minimum quickly, even 
if the resulting bracket is wide. We chose to increase the step size by a constant factor; 
that is, we use hj + 1 = c/t;, c > 1 . 

Golden Section Search 

The golden section search is the counterpart of bisection used in finding roots of 
equations. Suppose that the minimum of /(x) has been bracketed in the interval 
(a, b) of length h. To telescope the interval, we evaluate the function at x\ = b — Rh 
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and x 2 = a+ Rh, as shown in Fig. 10.2(a). The constant R vvill be determined shortly. 
I f /i > / 2 as indicated in the figure, the minimum lies in (x t , b); otherwise it is located 
in {a, x 2 ). 



(a) 


Figure 10.2. Golden section telescoping. 


f(x) 







Rh' _j 



U_ Rh 1 





a 

Xi x 2 

b 



h 1 




(b) 


Assuming that /1 > ( 2 , we seL a <- X\ and X\ x 2 , which yields a new interval [a. b) ol’ 
length h' = Rh, as illustrated in l ; ig. 10.2(b).Tocarryoutthenexttelescopingoperation 
we evaluate the function at x 2 = a + Rh and repeat the process. 

The procedure works only if Figs. 10.1 (a) and (b) are similar; i.e., if the same 
constant R locates Xi and x 2 in both figures. Referring to Fig. 10.2(a), we note that 
x 2 — X\ = 2 Rh — h. The same distance in Fig. 10.2(b) is X\ - a = h! - Rh'. Equating 
the two, we get 


2 Rh - h=h' - Rh’ 

Substituting h! = Rh and cancelling h yields 

2R- 1 = R(1 - R) 

the solution of which is the golden ratio? 4 : 

R = -1 + s/5 = Q 618 Q33 989 ( 10 .3) 

2 

Note that each telescoping decreases the interval containing the minimum by the 
factor R, which is not as good as the factor is 0.5 in bisection. However, the golden 
search method achieves this reduction with one function evaluation, whereas two 
evaluations would be needed in bisection. 


24 R is the ratio of the sides of a “golden rectangle,” considered by ancient Greeks to have the perfect 
proportions. 
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The number of telescoping required to reduce h from \b— a\ to an error tole- 
rance e is given by 


\b-a\ R n = e 


which yields 


ln(e/ | b— a\) 
\nR 


-2.078 087 ln-- 

| b-a\ 


(10.4) 


■ goldSearch 

This module contains the brackedng and the golden secdon search algorithms. For 
the factor that multiplies successive search intervals in bracket we chose c = 1 + R. 

## module goldSearch 

’’’ a,b = bracket(f,xStart,h) 

Finds the brackets (a,b) of a minimum point of the 
user-supplied scalar function f(x). 

The search starts downhill from xStart with a step 
length h. 

x,fMin = searchff,a,b,tol=l.Oe-6) 

Golden section method for determining x that minimizes 
the user-supplied scalar function f(x). 

The minimum must be bracketed in (a,b). 

from math import log 

def bracket(f,xl,h): 
c = 1.618033989 
fl = f(xl) 

x2 = xl + h; f2 = f(x2) 

# Determine downhill direction and change sign of h if needed 
if f2 > fl: 

h = -h 

x2 = xl + h; f2 = f(x2) 

# Check if minimum between xl - h and xl + h 
if f2 > f1: return x2,xl - h 

# Search loop 

for i in range (100): 
h = c*h 
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x3 = x2 + h; f3 = f(x3) 
if f3 > f2: return xl,x3 
xl = x2; x2 = x3 
f 1 = f 2 ; f 2 = f 3 

print ’’Bracket did not find a mimimum’’ 

def search(f,a,b,tol=l.Oe-9) : 

nlter = -2.078087*log(tol/abs(b-a)) # Eq. (10.4) 

R = 0.618033989 
C = 1.0 - R 

# First telescoping 

xl = R*a + C*b; x2 = C*a + R*b 
fl = f(xl); f2 = f(x2) 

# Main loop 

for i in range(nlter): 
if fl > f2: 
a = xl 

xl = x2; fl = f2 
x2 = C*a + R~b; f2 = f(x2) 
else : 

b = x2 

x2 = xl; f2 = fl 
xl = R*a + C*b; fl = f(xl) 
if f1 < f2: return xl,f1 
else: return x2,f2 

EXAMPLE 10.1 

Use goldSearch to find x that minimizes 

f(x) = 1.6x 3 + 3X 2 — 2x 

subject to the constraint x > 0. Compare the resuit with the analytical solution. 

Solution This is a constrained minimization problem. The minimum of /(x) is either 
a stationary point in x > 0, or located at the constraint boundary x = 0. We handle 
the constraint with the penalty function method by minimizing /(x) + X [min(0,x)] 2 . 

Starting at x = 1 and choosing h= 0.01 for the first step size in bracket (both 
choices being rather arbitrary), we arrive at the following program: 

#!/usr/bin/python 
## examplel0_l 
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from goldSearch import * 
def f(x): 

lam =1.0 # Constraint multiplier 

c = min(0.0, x) # Constraint function 
return 1.6*x**3 + 3.0*x**2 - 2.0*x + lam*c**2 

xStart = 1.0 
h = 0.01 

xl,x2 = bracket(f,xStart,h) 
x,fMin = search(f,xl,x2) 
print ’’x =’ ’ , x 
print ’’f(x) =’’,fMin 

raw_input (' '\nPress return to exit’’) 

The resuit is 

x = 0.27349402621 
f (x) = -0.28985978555 

Since the minimum was found to be a stationary point, the constraint was not 
active. Therefore, the penalty function was superfluous, but we did not know that at 
the beginning. 

The locations of stationary points are obtained analytically by solving 
f\x) = 4.8x 2 + 6x —2 = 0 

The positive root of this equation is x = 0.273 494. As this is the only positive root, 
there are no other stationary points in x > 0 that we must check out. The only other 
possible location of a minimum is the constraint boundary x = 0. But here /(0) = 0 
is larger than the function at the stationary point, leading to the conclusion that the 
global minimum occurs at x = 0.273 494. 

EXAMPLE 10.2 



x 
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The trapezoid shown is the cross section of a beam. It is formed by removing the top 
from a triangle of base B = 48 mm and height H = 60 mm. The problem is the find 
the height y of the trapezoid that maximizes the section modulus 

S = Ix/c 

where 7* is the second moment of the cross-sectional area about the axis that passes 
through the centroid C of the cross section. By optimizing the section modulus, we 
minimize the maximum bendingstress CT max = M/S in the beam, Mbeingthebending 
moment. 

Solution Considering the area of the trapezoid as a composite of a rectangle and 
two triangles, we find the section modulus through the following sequence of 
computations: 


Base of rectangle 

a= B(H - y )/H 

Base of triangle 

b = {B — a) /2 

Area 

A = (B+ a) y/2 

First moment of area about x-axis 

Qx = ( ay ) y/2 + 2 (by/ 2) y/3 

Location of centroid 

d = Q x /A 

Distance involved in S 

c=y-d 

Second moment of area about x-axis 

I x = af/3 + 2 (bf/12) 

Parallel axis theorem 

% 

1 

II 

-3* 

Section modulus 

II 

C/D 


We could use the formulas in the table to derive S as an explicit function of y, but 
that would involve a lot of error-prone algebra and resuit in an overly complicated 
expression. It makes more sense to let the computer do the work. 

The program we used and its output are listed below. As we wish to maximize S 
with a minimization algorithm, the merit function is — S. There are no constraints in 
this problem. 

#!/usr/bin/python 

## examplel0_2 

from goldSearch import * 

def f(y): 

B = 48.0 
H = 60.0 
a = B*(H - y)/H 
b = (B - a)/2.0 
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A = (B + a) *y/2.0 
Q = (a*y**2)/2.0 + (b*y**2)/3.0 
d = Q/A 
c = y - d 

I = (a*y**3)/3.0 + (b*y**3)/6.0 
Ibar = I - A*d* * 2 
return -Ibar/c 

yStart =60.0 # Starting value of y 

h = 1.0 # Size of first step used in bracketing 

a,b = bracket(f,yStart,h) 

yOpt,fOpt = search(f,a,b) 

print ’'Optimal y =’’,yOpt 

print ’'Optimal S =’’,-fOpt 

print ’’S of triangle =’’,-f(60.0) 

raw_input(’’Press return to exit’’) 

Optimal y = 52.1762738732 
Optimal S = 7864.43094136 
S of triangle = 7200.0 

The printout includes the section modulus of the original triangle. The optimal 
section shows a 9.2% improvement over the triangle. 


10.3 Conjugate Gradient Methods 
Introduction 

We now look at optimization in n-dimensional design space. The objective is to min- 
imize -F(x), where the components of x are the n independent design variables. One 
way to tackle the problem is to use a succession of one-dimensional minimizations 
to close in on the optimal point. The basic strategy is 

• Choose a point xo in the design space. 

• loop with i= 1,2,3,... 

Choose a vector v,-. 

Minimize F[x) along the line through x,_i in the direction of v,-. Let the mini¬ 
mum point be x,-. 

if |x/ — Xj_i| < e exit loop 


end loop 
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The minimization along a line can be accomplished with any one-dimensional 
optimization algorithm (such as the golden section search). The only question left 
open is how to choose the vectors v*. 


Conjugate Directions 

Consider the quadratic function 

F (x) = c — biXi + ^ 

i i j 

= c- b r x+ ^x r Ax (10.5) 

Differentiation with respect to jc,- yields 

dl i ~~ bi + ^ ijXi 

which can be written in vector notation as 


VF = —b + Ax 


( 10 . 6 ) 


where VF 7 is the gradient of F. 

Now consider the change in the gradient as we move from point xo in the direction 
of a vector u. The motion takes place along the line 

X = Xo + su 

where s is the distance moved. Substitution into Eq. (10.6) yields the expression for 
the gradient at x: 


= -b + A(xo + su) = VF\^ + s Au 

Note that the change in the gradient is 5 Au. If this change is perpendicular to a vector 
v; that is, if 


v r Au = 0 (10.7) 

the directions of u and v are said to be mutually conjugate (noninterfering). The 
implication is that once we have minimized F (x) in the direction of v, we can move 
along u without ruining the previous minimization. 

For a quadratic function of n independent variables it is possible to construet n 
mutually conjugate directions. Therefore, it would take precisely n line minimizations 
along these directions to reach the minimum point. If F (x) is not a quadratic function, 
Eq. (10.5) can be treated as a local approximation of the merit function, obtained by 
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truncating the Taylor series expansion of F (x) about xo (see Appendix AI): 

F(x) » F{x o) + VF(xo)(x-Xo) + ^(x - Xo^HCxoHx - Xo) 

Now the conjugate directions based on the quadratic form are only approximations, 
valid in the close vicinity of xo. Consequently, it would take several cycles of n line 
minimizations to reach the optimal point. 

The various conjugate gradient methods use different techniques for constructing 
conjugate directions. The so-called zero-order methods work with F[x) only whereas 
the first-order methods utilize both F(x) and VF. The first-order methods are com- 
putationally more efficient, of course, but the input of V F, if it is available at ali, can 
be very tedious. 

PoweM's Method 

PowelPs method is a zero-order method, requiring the evaluation of F (x) only. If the 
problem involves ndesign variables, the basic algorithm is 

• Choose a point xo in the design space. 

• Choose the starting vectors v,-, i = 1, 2,..., n (the usual choice is v,- = e,-, where e,- 
is the unit vector in the x,-coordinate direction). 

• cycle 


do with i = 1, 2 ,..., n 

Minimize F[x) along the line through x,_i in the direction of v,. Let the 
minimum point be x,-. 

end do 

V/z+l Xo x„ 

Minimize F (x) along the line through xo in the direction of v n+ \. Let the 
minimum point be x„ + 1 . 

if |x„ + i - Xo| < e exit loop 
do with i = 1,2,... ,n 

V; <- V; + i (vi is discarded, the other vectors are reused) 
end do 

• end cycle 

Powell demonstrated that the vectors v, i+ i produced in successive cycles are mu- 
tually conjugate, so that the minimum point of a quadratic surface is reached in 
precisely n cycles. In practice, the merit function is seldom quadratic, but as long as 
it can be approximated locally by Eq. (10.5), PowelLs method will work. Of course, it 
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usually takes more than n cycles to arrive at the minimum of a nonquadratic function. 
Note that it takes «line minimizations to construet each conjugate direction. 

Figure 10.3 (a) illustrates one typical cycle of the method in a two dimensional 
design space [n = 2). We start with point xo and vectors vi and v 2 . Then we find the 
distance S| that minimizes F(x 0 + svi), finishing up at point xi = xo + .s'i V] . Next, we 
determine s 2 that minimizes F(xj + ,s'v 2 ) which takes us to x 2 = xi + s 2 v 2 . The last 
search direction is v 3 = x 2 — xo. After finding 53 by minimizing F(x 0 + sv 3 ) we get to 
x 3 = xq + s 3 v 3 , completing the cycle. 



Figure 10.3(b) shows the moves carried out in two cycles superimposed on the 
contour map of a quadratic surface. As explained before, the first cycle starts at point 
P 0 and ends up at P 3 . The second cycle takes us to P s , which is the optimal point. The 
directions P 0 P 3 and P 3 P 6 are mutually conjugate. 

PowelFs method does have a major flaw that has to be remedied—if F(x) is not 
a quadratic, the algorithm tends to produce search directions that gradually be- 
come linearly dependent, thereby ruining the progress towards the minimum. The 
source of the problem is the automatic discarding of vi at the end of each cycle. It 
has been suggested that it is better to throw out the direction that resulted in the 
largest decrease of P (x), a policy that we adopt. It seems counterintuitive to discard 
the best direction, but it is likely to be close to the direction added in the next cycle, 
thereby contributing to linear dependence. As a resuit of the change, the search di¬ 
rections cease to be mutually conjugate, so that a quadratic form is not minimized 
in n cycles any more. This is not a significant loss since in practice F (x) is seldom a 
quadratic. 

Powell suggested a few other refinements to speed up convergence. Since they 
complicate the bookkeeping considerably, we did not implement them. 
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■ powell 

The algorithm for Powell’s method is listed below. It utilizes two arrays: df contains 
the decreases of the merit function in the first n moves of a cycle, and the matrix u 
Stores the corresponding direction vectors v,- (one vector per row). 

## module powell 

’’’ xMin,nCyc = powell(E,x,h=0.1,tol=l.Oe-6) 

Powell's method of minimizing user-supplied function F(x). 
x = starting point 

h = initial search increment used in 'bracket' 
xMin = mimimum point 
nCyc = number of cycles 

from numarray import identity,array,dot,zeros,Float64,argmax 
from goldSearch import * 
from math import sqrt 

def powell(F,x,h=0.1,tol=l.Oe-6): 

def f(s): return F(x + s*v) 

n = len(x) 

df = zeros((n),type=Float64) 
u = identity(n)*1.0 
for j in range(30): 
xOld = x.copyO 
fOld = F(xOld) 

# First n line searches re 
for i in range(n): 

v = u[ i] 

a,b = bracket(f,0.0,! 
s,fMin = search(f,a, 
df[i] = fOld - fMin 
fOld = fMin 
x = x + s*v 

# Last line search in the cycle 
v = x - xOld 
a,b = bracket(f,0.0,h) 
s,fLast = search(f,a,b) 


# F in direction of v 

# Number of design variables 

# Decreases of F stored here 

# Vectors v stored here by rows 

# Allow for 30 cycles: 

# Save starting point 

ord decreases of F 


x 


x + s*v 
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# Check for convergence 

if sqrt(dot(x-xOld,x-xOld)/n) < tol: return x,j+l 

# Identify biggest decrease & update search directions 
iMax = int(argmax(df)) 

for i in range(iMax,n-1): 
u[i] = u[i+l] 
u[n-l] = v 

print ’'Powell did not converge’’ 

EXAMPLE 10.3 

Find the minimum of the function 25 

F = 100(y — x 2 ) 2 + (1 — x) 2 

with Powell’s method starting at the point (—1,1). This function has an interesting 
topology. The minimum value of F occurs at the point (1,1). As seen in the figure, 
there is a hump between the starting and minimum points which the algorithm must 
negotiate. 



Solution The program that solves this unconstrained optimization problem is 

#!/usr/bin/python 
## examplel0_3 
from powell import * 
from numarray import array 

def F(x): return 100.0*(x[l] - x[0]**2)**2 + (1 - x[0])**2 

xStart = array([-1.0, 1.0]) 
xMin,nIter = powell(F,xStart) 
print '’x =’’,xMin 

25 FromShoup.T. E., andMistree, F., Optimization Methods with Applicationsfor Personal Computers, 
Prentice-Hall, 1987. 
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print ’’F(x) =’’,E(xMin) 

print ’'Number of cycles =’’,nlter 

raw_input (’'Press return to exit’’) 

As seen in the printout, the minimum point was obtained in 14 cycles. 

X = [ 1. 1.] 

F(x) = 4.93038065763e-030 
Number of cycles = 14 

EXAMPLE 10.4 

Use powell to determine the smallest distance from the point (5, 8) to the curve 
xy=5. 

Solution This is a constrained optimization problem: minimize F{x , y) = {x — 5) 2 + 
( y — 8) 2 (the square of the distance) subject to the equality constraint xy - 5 = 0. The 
following program uses Powells method with penalty function: 

#!/usr/bin/python 
## examplel0_4 
from powell import * 
from numarray import array 
from math import sqrt 

def F(x): 

lam =1.0 # Penalty multiplier 

c=x[0]*x[l]-5.0 # Constraint equation 

return distSq(x) + lam*c**2 # Penalized merit function 

def distSq(x): return (x[0] - 5)**2 + (x[l] - 8)**2 

xStart = array([1.0, 5.0]) 

x,numlter = powell(F,xStart,0.01) 

print ’’Intersection point =’’,x 

print ’'Minimum distance =’’, sqrt(distSq(x)) 

print ’’xy x[0]*x[l] 

print ’'Number of cycles =’’,numlter 

raw_input (’'Press return to exit’’) 

As mentioned before, the value of the penalty function multiplier 1 (called lam 
in the program) can have profound effects on the resuit. We chose X = 1 (as in the 




396 


Introduction to Optimization 


program listing) with the following resuit: 

Intersection point = [ 0.73306759 7.58776399] 

Minimum distance = 4.28679959441 
xy = 5.56234382324 
Number of cycles = 6 

The small value of A favored speed of convergence over accuracy. Since the viola- 
tion of the constraint xy = 5 is clearly unacceptable, we ran the program again with 
X = 10 000 and changed the starting point to (0.733 07, 7.587 76), the end point of the 
first run. The results shown below are now acceptable: 

Intersection point = [ 0.65561312 7.6265359 ] 

Minimum distance = 4.36040970941 
xy = 5.00005696388 
Number of cycles = 5 

Could we have used X = 10 000 in the first run? In this case we would be lucky 
and obtain the minimum in 19 cycles. Hence we save eight cycles by using two runs. 
However, a large X often causes the algorithm to hang up, so that it generally wise to 
start with a small X. 

Fletcher-Reeves Method 

Let us assume again that the merit function has the quadratic form in Eq. (10.5). Given 
a direction v, it took Powelfs method n line minimizations to construet a conjugate 
direction. We can reduce this to a single line minimization with a first-order method. 
Here is the procedure, known as the Fletcher-Reeves method: 

• Choose a starting point xo. 

• go <-VF(xo) 

• v 0 <— go (lacking a previous search direction, we choose the steepest descent). 

• loop with i= 0 , 1 , 2 ,... 

Minimize F (x) along v,; let the minimum point be x,- + i. 
g;+t <- —VF(x m ). 

if |g/+i| < e or | /-'(xz+i) — F(x,)| < £ exit loop (convergence criterion). 

Y (gz+1 ' gi+l)/(gi' ' g/)• 
v;+i g;+i + /Vi. 


• endloop 

It can be shown that v, and v,- + i are mutually conjugate; that is, they satisfy the 
relationship vf Avj+i = 0 . Also g, ■ g i+ i = 0. 
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The Fletcher-Reeves method will find the minimum of a quadratic function in 
n iterations. If F (x) is not quadratic, it is necessary to restart the process after every 
n iterations. A variant of the Fletcher-Reeves method replaces the expression for y by 

Y = (g/+1 ~ g,) - g/+1 (10.6) 
Si ' Si 

For a quadratic F (x) this change makes no difference since g, and g, +1 are orthogonal. 
Fiowever, for merit functions that are not quadratic, Eq. (10.6) is claimed to eliminate 
the need for a restart after «iterations. 

■ fletcherReeves 
## module fletcherReeves 

’’’ xMin,nIter = optimize(F,gradF,x,h=0.01,tol=l.Oe-6 
Fletcher-Reeves method of minimizing a function. 

F(x) = user-supplied function to be minimized. 

gradF(x) = user-supplied function for grad(F). 

x = starting point. 

h = initial search increment used in 'bracket’. 

xMin = mimimum point. 

nlter = number of iterations. 


from numarray import array,zeros,Float64,dot 
from goldSearch import * 
from math import sqrt 


def optimize(F,gradF,x,h=0.1,tol=l.Oe-6): 

def f(s): return F(x + s*v) # Line function along v 

n = len(x) 
gO = -gradF(x) 
v = gO.copyO 
F0 = F(x) 

for i in range(200): 

a,b = bracket(f,0.0,h) # Minimization along 

s,fMin = search(f,a,b) # a line 
x = x + s*v 
FI = F(x) 
gl = -gradF(x) 

if (sqrt(dot(gl,gl)) <= tol) or (abs(F0 - FI) < tol): 
return x,i+l 
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gamma = dot((gl - gO),gl)/dot(gO,gO) 
v = gl + gamma*v 
gO = gl.copyO 
FO = FI 

print ’'fletcherReeves did not converge’’ 

EXAMPLE 10.5 

Use the Fletcher-Reeves method to locate the minimum of 


F(x ) = 10x 2 + 3 xf — 10xiJC 2 + 2*i 

Start with x 0 = 0 0 j . 


Solutiori Since F(x) is quadratic, we need only two iterations. The gradient of F is 


First iteration: 


VF(x) = 


dF/dXi 


20x\ — 10x2 H - 2 

dF/dX2 


— 10xi + 6 x 2 



-2 


-2 


—2s 

go = -VF(xo) = 

0 

II 

cra 

O 

II 

0 

Xo + sv 0 = 

0 


fis) = F{x o + svq) = 10(2 s) 2 + 3(0) 2 - 10(-2s)(0) + 2(-2s) 
= 40s 2 — 4s 


fis) = 80s - 4 = 0 


s = 0.05 


Xi = Xo + 5V 0 



+ 0.05 



- 0.1 

0 


Second iteration: 


gi = -VF(xi) 


" —20(—0.1) + 10(0) - 2 


0 

1 

1— 1 

O 

T 

o 

I— 1 

1 

o 


-1.0 


y = iLl8l 

go • go 


1.0 


0.25 


vi = gi + yv 0 


0 

- 1.0 


+ 0.25 



-0.5 

- 1.0 


-0.1 

+ s 

"—0.5 _ 


-0.1 - 0.5s 

0 


-1.0 


-s 


Xi + SVi = 
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/(5) = F(x 1 + SVi) 

= 10(—0.1 - 0.5s) 2 + 3(-s) 2 - 10(—0.1 - 0.5s)(—s) + 2(-0.1 - 0.5s) 
= 0.5s 2 - s - 0.1 


f(s) = s- 1 = 0 5=1.0 


X 2 = Xi + SVi 


- 0.1 

0 


+ 1.0 


-0.5 

- 1.0 


- 0.6 

- 1.0 


We have now reached the minimum point. 

EXAMPLE 10.6 



The figure shows the cross section of a channel carrying water. Determine h, b 
and 9 that minimize the length of the wetted perimeter while maintaining a cross- 
sectional area of 8 m 2 . (Minimizing the wetted perimeter results in least resistance to 
the flow.) Use the Fletcher-Reeves method. 


Solution The cross-sectional area of the channel is 

A= ^ [b+ (h+2htan0)] h= (b+ htan9)h 
and the length of the wetted perimeter is 

S = b+ 2{hsec9) 

The optimization problem can be cast as 

minimize b+ 2hsec9 
subject to {b+ htan8)h= 8 

Equality constraints can often be used to eliminate some of the design variables. 
In this case we can solve the area constraint for b, obtaining 

, 8 , 
b = - - h tand 
h 

Substituting the resuit into the expression for S, we get 


g 

S = - - h tan$ + 2hsec9 
h 
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We have now arrived at an unconstrained optimization problem of finding h and 
8 that minimize S. The gradient of the merit function is 


dS/dh 


—8/ /z 2 — tan 8 + 2 sec 8 

8S/88 


— hsec 2 8 + 2hsec8tand 


Letting x = yh 8 J and starting with xo = y 2 0 J , we arrive at the following program: 

#!/usr/bin/python 

## examplel0_6 

from fletcherReeves import * 

from numarray import array,zeros,Float64 

from math import cos,tan,pi 

def F(x): 

return 8.0/x[0] - x[0]*(tan(x[1]) - 2.0/cos(x[1])) 
def gradF(x): 

g = zeros((2),type=Float64) 

g[0] = -8.0/(x[0]**2) - tan(x[l]) + 2.0/cos(x[l]) 
g[l] = x[0]*(-1.0/cos(x[l]) + 2.0*tan(x[l]))/cos(x[1]) 
return g 

x = array([2.0, 0.0]) 
x.nlter = optimize(F,gradF,x) 
b = 8.0/x[0] - x[0]*tan(x[l]) 
print ’’h =’’,x[0],’’m’’ 
print ’’b =’’,b,’'m'’ 

print ’'theta =’ ’ ,x[1]*180.0/pi, ’ 'deg' ’ 
print ''perimeter F(x),’’m’’ 

print ''Number of iterations =’’,nlter 
raw_input(’’Press return to exit’’) 

The results are: 

h = 2.14914172295 m 
b = 2.48162366149 m 
theta = 29.9997001208 deg 
perimeter = 7.44483887289 m 
Number of iterations = 4 
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PROBLEM SET 10.1 

1. ■ The Lennard-Jones potential between two molecules is 


V= 4e 



6 ' 


where e and a are constants, and r is the distance between the molecules. Use 
the module goldSearch to find a /r that minimizes the potential and verify the 
resuit analytically. 

2. ■ One wave function ofthe hydrogen atom is 


f = C (27 - 18rr + 2er 2 ) e~ a/3 


where 


a = zr/ao 


C = 




2/3 


z = nuclear charge 
flo = Bohr radius 
r = radial distance 


Find a where \!/ is al a minimum. Verify the resuit analytically. 

3. ■ Determine the parameter p that minimizes the integral 



sin x cos pxdx 


Hint: use numerical quadrature to evaluate the integral. 


4. ■ 


R,= 2Q R 2 = 3.6 U 


-AM/V- 

--WWV- 

u . 

;* 


i R 


kj 

-MW- 

-MW- 

R 3 = 1.5 0 

f? 4 =1.8Q 


Kirchoffs equations for the two loops of the electrical Circuit are 

Riii + Rah + Rtfi — 4 ) = E 
Rik + Rih + R 5 k + R(.h — h) = 0 


Find the resistance R that maximizes the power dissipated by R. Hint: solve 
Kirchoffs equations numerically with one of the functions in Chapter 2. 
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5. ■ 



A wire carrying an electric current is surrounded by rubber insulation of outer 
radius r. The resistance of the wire generates heat, which is conducted through 
the insulation and convected into the surrounding air. The temperature of the 
wire can be shown to be 


T 


q_ ( \n{r/a) J_\ 

2n\k hr) 00 


where 


q = rate of heat generation in wire = 50 W/m 
a = radius of wire = 5 mm 

k = thermal conductivity of rubber = 0.16 W/m • K 
h = convective heat-transfer coefficient = 20 W/m 2 • K 
Too = ambient temperature = 280 K 

Find r that minimizes T. 

6 . ■ Minimize the function 

F(x, y) = {x- l) 2 + (y- l) 2 

subject to the constraints x + y < 1 and x > 0.6. 

7. ■ Find the minimum of the function 

F{x, y) = 6x 2 + y 3 + xy 

in y > 0. Verify the resuit analytically. 

8 . ■ Solve Prob. 7 if the constraint is changed to y > —2. 

9. ■ Determine the smallest distance from the point (1,2) to the parabola y = x 2 . 




























































403 


10.3 Conjugate Gradient Methods 


10. ■ 




0.2 m 


nr 

X 

1 

0.4 m 

C *-T 

d 

J 



0.4 m 


Determine x that minimizes the distance d between the base of the area shown 
and its centroid C. 


11 . ■ 



The cylindrical vessel of mass M has its center of gravity at C. The water in the 
vessel has a depth x. Determine x so that the center of gravity of the vessel-water 
combination is as low as possible. Use M= 115 kg, H = 0.8 m and r = 0.25 m. 


a 



—i 

b 

L 


—b~ 







The sheet of cardboard is folded along the dashed lines to form a box with an 
open top. If the volume of the box is to be 1.0 m 3 , determine the dimensions a 
and fothat would use the least amount of cardboard. Verify the resuit analytically. 
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13. ■ 



The elastic cord ABC has an extensional stiffness k. When the vertical force P is 
applied at B, the cord deforms to the shape ABC. The potential energy of the 
system in the deformed position is 


V= -Pv + 


k(a + b) 2 k[a + b) 2 
~2^~' AB+ ^^‘ BC 


where 


8 AB = \/(a+ u) 2 + v 2 - a 
S B c = V(b- u) 2 + v 1 - b 

are the elongations of AB and BC. Determine the displacements u and v by min- 
imizing V (this is an application of the principle of minimum potential energy: 
a system is in stable equilibrium if its potential energy is at a minimum). Use 
a = 150 mm, b= 50 mm, k = 0.6 N/mm and P = 5 N. 



Each member of the truss has a cross-sectional area A. Find A and the angle 9 
that minimize the volume 

V=^_ 

cos e 

of the material in the truss without violating the constraints 


a < 150 MPa 8 < 5 mm 
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where 


S = 


2Asin8 
Pb 


= stress in each member 

= displacement at the load P 


2EAsin28 sin 0 
and E = 200 x 10 9 P a . 

15. ■ Solve Prob. 14 if the allowable displacement is changed to 2.5 mm. 

16. ■ 

Lh i r„ 


L= 1.0 m 


[=1.0m 


P= 10 kN 


The cantilever beam of circular cross section is to have the smallest volume pos- 
sible subject to constraints 


where 


«ji<180MPa o- 2 <180MPa 5 < 25 mm 


8PL ■ • , « 

a i = — 5 - = maximum stress m left hall 
nrf 


APL 


02 


nri 


= maximum stress in right half 


APL 3 /7 1 \ 

S = -(— + — )= displacement at free end 

3j tE \r* r|/ 

and E = 200 GPa. Determine r\ and r 2 . 

17. ■ Find the minimum of the function 

F {x, y , z) = 2x 2 + 3y 2 + z 2 + xy + xz — 2y 

and confirm the resuit analytically. 

18. ■ 



The cylindrical Container has a conical bottom and an open top. If the volume V 
of the Container is to be 1.0 m 3 , find the dimensions r, h and b that minimize the 
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surface area S. Note that 


V = 7t r 2 ( - + h 


3 


S = nr(2h + y/ bP + r 2 ^j 


19. ■ 


4 i 





© 


P = 200 kN 


t P= 200 kN 

The equilibrium equations of the truss shown are 

4 3 

tri-Ai + — (J 2 A 2 = P — O 2 A -2 + <^3^3 = P 

5 5 

where a i is the axial stress in member i and At are the cross-sectional areas. 

The third equation is supplied by compatibility (geometrical constraints on the 
elongations of the members): 


16 


9 


—ot - 5 er 2 + -0-3 = 0 
o o 


Find the cross-sectional areas of the members that minimize the weight of the 
truss without the stresses exceeding 150 MPa. 


20. ■ 



A cable supported at the ends carries the weights Wi and IV 2 . The potential energy 
of the system is 


V = — Wiyi — W 2 yz 

= —W\L\ sin^i — W 2 (L\ sin0i + L 2 sin d 2 ) 
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10.4 Other Methods 


and the geometric constraints are 

L\ cos + L 2 cos 0 2 + L 3 cos 9 3 = B 
L\ sin0i + L 2 sin0 2 + L 3 sin0 3 = H 

The principle of minimum potential energy States that the equilibrium configu- 
ration of the system is the one that satisfies geometric constraints and minimizes 
the potential energy. Determine the equilibrium values of 0i, 0 2 and 0 3 given 
that Li = 1.2 m, L 2 = 1.5 m, L 3 = 1.0 m, B = 3.5 m, H = 0, Wi = 20 kN and 
W 2 = 30 kN. 


10.4 Other Methods 

The Nelder-Mead method, also know as the downhill simplex algorithm, is a popular 
and robust method of optimization. Its main attraction is a common sense geometrical 
approach that requires no mathematical sophistication. In speed of execution of the 
downhill simplex is not competitive with PowelTs method. 

Simulated annealing methods have been successfully employed for complex 
problems involving many design variables. These methods are based on an anal- 
ogy with the annealing as a slowly cooled liquid metal solidifies into a crystalline, 
minimum energy structure. One distinguishing feature of simulated annealing is its 
ability to pass over local minima in its search for the global minimum. 

A topic that we reluctantly omitted is the simplex method of linear programming. 
Linear programming deals with optimization problems where the merit function and 
the constraints are linear expressions of the independent variables. The general linear 
programming problem is to minimize the objective function 

n 

i 7 = E a ' x ‘ 

i= 1 


n 


E Bi i x i < 

i= 1,2,. 

.., m\ 

j= i 



n 



E FijXj > d, 

i = 1,2,. 

• •, m 2 

7=1 



n 



'y ] DijXj — di, 

i= 1,2,. 

..,m 3 

7=1 



Xi > 0, 

i= 1,2,.. 

. n 


subject to the constraints 
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where the constants bi, Ci and 4: are nonnegative. The roots of linear programming 
lie in cost analysis, operations research and related fields. We skip this topic because 
there are very few engineering applications that can be formulated as linear program¬ 
ming problems. In addition, a fail-safe implementation of the simplex method results 
in a rather complicated algorithm. This not to say that the simplex method has no 
place nonlinear optimization. There are several effective methods that rely in part on 
the simplex method. For example, problems with nonlinear constraints can often be 
solved by a piecewise application of linear programming. The simplex method is also 
used to compute search directions in the so-called method offeasible directions. 




Appendices 


AI Taylor Series 

Function of a Single Variable 

The Taylor series expansion of a function /(x) about the point x = a is the infinite 
series 


/(x) = /(a) + f(a){x- a) + f\a) ^ + f"{a ) tX ^ H- 


(AI) 


In the special case <3=0 the series is also known as the MacLaurin series. It can be 
shown that the Taylor series expansion is unique in the sense that no two functions 
have identical Taylor series. 

A Taylor series is meaningful only if ali the derivatives of /(x) exist at x = a and 
the series converges. In general, convergence occurs only if x is sufficiently close to a; 
i.e.,if|x — a\ < e, where eis called the radius of convergence. In many cases s is infinite. 

Another useful form of the Taylor series is the expansion about an arbitrary value 
ofx: 



(A2) 


Since it is not possible to evaluate ali the terms of an infinite series, the effect of 
truncating the series in Eq. (A2) is of great practical importance. Keeping the first 
n + 1 terms, we have 


/(x + h) = /(x) + f(x)h+ /'(x) — H-f f in> (x) — + E n 


(A3) 


where E n is the truncation error (sum of the truncated terms). The bounds on the 
truncation error are given by Taylor’s theorem : 



(A4) 
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where § is some point in the interval (x, x + h) . Note that the expression for E n is 
identical to the first discarded term of the series, but with x replaced by $. Since the 
value of f is undetermined (only its limits are known), the most we can get out of 
Eq. (A4) are the upper and lower bounds on the truncation error. 

If the expression for is not available, the information conveyed by 

Eq. (A4) is reduced to 


E n = 0{h n+1 ) 


(A5) 


which is a concise way of saying that the truncation error is of the order of h" +1 , or 
behaves as h" +1 . If h is within the radius of convergence, then 


0{h' 1 ) > Oih!^ 1 ) 


i.e., the error is always reduced if a term is added to the truncated series (this may not 
be true for the first few terms). 

In the special case n = 1, Taylor’s theorem is known as the rnean value theorem: 


f(x + h) = /(x) + f'{f)h, x < £ < x + h 


(A6) 


Function of Several Variables 

If / is a function of the m variables xi, X 2 ,..., x m , then its Taylor series expansion 
about the point x = [xi, xz, ..., x m ] T is 



hihj + • • • 


(A7) 


This is sometimes written as 


/(x + h) = /(x) + V /(x) ■ h + ih r H(x)h H- 


(A8) 


The vector V / is known as the gradient of / and the matrix H is called the Hessian 
matrix of /. 


EXAMPLE AI 

Derive the Taylor series expansion of /(x) = ln(x) about x = 1. 


Solution The derivatives of / are 



Evaluating the derivatives at x = 1, we get 

/(1) = 1 f"( 1) = -1 f"( 1) = 2! / t4) (l) = -3! etc. 
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which upon substitution into Eq. (AI) together with a = 1 yields 

(X - l) 2 (X - l) 3 (X - l) 4 

ln(x) = 0 + (x- 1) - +2! -3! .. +• 


2 ! 


3! 


4! 


= (x- 1)- i(x- 1) 2 + i(x- l) 3 - i(x- 1) 4 + - 


EXAMPLE A2 

Use the first five terms of the Taylor series expansion of e* about x = 0 

y X 2 X 3 X 4 

^“ 1 + X+ 2! + 3! + 4! + "' 
together with the error estimate to find the bounds of e. 

Solution 

1 1 1 „ 65 „ 

6 — 1 —1~ 1 —— -J- — 4~ — £4 = — £4 

2 6 24 24 


E i =?VW-=~- } 0 < f < 1 


/t 5 


5! 5! 

The bounds on the truncation error are 


(£4)min - 5 , - 12Q ( £ 4)max - 5 , - UQ 


Thus the lower bound on e is 


_ 65 

dn i n — ,, . 4“ 


163 


24 120 60 


and the upper bound is given by 


65 e max 
emax - Y\ + T20 


which yields 


Therefore, 


119 _ 65 

l20 emax ~ 24 


emax — 


325 

U9 


163 325 

- < e < - 

60 “ _ 119 


EXAMPLE A3 

Compute the gradient and the Hessian matrix of 

/(x, y) =ln Vx 2 + y 2 

at the point x = —2, y = 1. 
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Solution 

9/ _ 1 (1 2x \ x 9/ _ y 

dx yV + y 2 \2 J x 2 + y 2 ) x 2 + y 2 dy x 2 + y 2 

V /(x, y) = [ x/(x 2 + y 2 ) y/(x 2 + y 2 ) j 

V/(-2,1) = [-0.4 0.2 ] r 

9 2 / (x 2 + y 2 ) - x(2x) -x 2 + y 2 
9x 2 (x 2 + y 2 ) 2 (x 2 + y 2 ) 2 

9 2 / _ x 2 - y 2 
9y 2 (x 2 + y 2 ) 2 

9 2 / 9 2 ,/ ~2xy 

9x9y 9y9x (x 2 + y 2 ) 2 


H(x, y) 


-x 2 + y 2 
-2xy 



1 

(x 2 + y 2 ) 2 


H(—2,1) 


-0.12 0.16 

0.16 0.12 


A2 Matrix Algebra 

A matrix is a rectangular array of numbers. The size of a matrix is determined by the 
number of rows and columns, also called the dimensions of the matrix. Thus a matrix 
of m rows and n columns is said to have the size mx n (the number of rows is always 
listed first). A particularly important matrix is the square matrix, which has the same 
number of rows and columns. 

An array of numbers arranged in a single column is called a column vector, or 
simply a vector. If the numbers are set out in a row, the term row vector is used. Thus 
a column vector is a matrix of dimensions nx 1 and a row vector can be viewed as a 
matrix of dimensions 1 x n. 

We denote matrices by boldface, upper case letters. For vectors we use boldface, 
lower case letters. Flere are examples of the notation: 



1 

to 

CO 

_1 


b\ 

A = 

A 21 A 22 A 23 

b = 

h 


A 31 A 32 A 33 


h 


(A9) 
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Indices of the elements of a matrix are displayed in the same order as its dimensions: 
the rownumber comes first, followed by the column number. Only one indexis needed 
for the elements of a vector. 


Transpose 

The transpose of a matrix A is denoted by A r and defined as 


a t 

Aj 


= A } i 


The transpose operation thus interchanges the rows and columns of the matrix. If 
applied to vectors, it turns a column vector into a row vector and vice versa. For 
example, transposing A and b in Eq. (A9), we get 


A r 


An A21 A31 

A \2 A22 A32 

A 13 A 23 A 33 


b T = [hi h hi] 


An n x n matrix is said to be symmetric if A 7 ' = A. This means that the elements 
in the upper triangular portion (above the diagonal connecting An and A nn ) of a 
symmetric matrix are mirrored in the lower triangular portion. 


Addition 

The sum C = A + B of two m x «matrices A and B is defined as 

Qj = Aij + Bij, i = 1, 2,..., m; j = 1, 2,..., n (AIO) 

Thus the elements of C are obtained by adding elements of A to the elements of B. 
Note that addition is defined only for matrices that have the same dimensions. 


Multiplication 

The scalar or dot product c = a • b of the vectors a and b, each of size m, is defined as 

m 

c='Y^Okbk (Ali) 

k= 1 

It can also be written in the form c = a r b. 

The matrix product C = AB of an l x m matrix A and an m x n matrix B is 
defined by 

m 

Q; = ^ ^ AikBkj, 

k= 1 


i = 1,2,..., Z; j = 1,2,..., n 


(AI 2) 
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The definitiori requires the number of columns in A (the dimension m ) to be equal to 
the number of rows in B. The matrix product can also be defined in terms of the dot 
product. Representing the zth row of A as the vector a, and the j th column of B as the 
vector bj, we have 

Cy = a • bj (A 13 ) 


A square matrix of special importance is the identity or unit matrix 

'10 0 0 ' 

0 1 0 0 

I= 0 0 1 ■ 0 

_0 0 0 0 1 _ 

It has the property AI = IA = A. 


(A 14 ) 


Inverse 

The inverse of an n x n matrix A, denoted by A -1 , is defined to be an n x n matrix 
that has the property 

A~*A = A A -1 = I (AI 5 ) 


Determinant 


The determinant of a square matrix A is a scalar denoted by |A| or det(A). There is no 
concise definition of the determinant for a matrix of arbitrary size. We start with the 
determinant of a 2 x 2 matrix, which is defined as 


An Ai2 

A21 A22 


A\ 1 A22 — A12A21 


(A 16 ) 


The determinant of a 3 x 3 matrix is then defined as 


An 

A12 

A\3 

A21 

A22 

A23 

A31 

A32 

A33 


A22 

A23 

“ A\2 

A21 

A23 

+ a 13 ^ 21 

A22 

A32 

A33 

A31 

A33 

A31 

A32 


Having established the pattern, we can now define the determinant of an n x n 
matrix in terms of the determinant of an [n — 1) x (n — 1) matrix: 


l A l = ^(-l) fc+1 Ai fc Mi fc 
k= 1 


(AI 7 ) 

















415 


A2 Matrix Algebra 


where M ] k is the determinant of the (n — 1) x (n— 1) matrix obtained by deleting the 
/Lh rowand fcth column of A. The term (— \) k+l Mfk is called a cofactor of A,*. 

Equation (Ai 7) is known as Laplace’s development of the determinant on the 
first row of A. Actually Laplace's development can take place on any convenient row. 
Choosing the /ili row, we have 

n 

i a i = J2 { - l)k+iAikMik (A18) 

*!=i 

The matrix A is said to be singular if |A| = 0. 


Positive Definiteness 


An n x n matrix A is said to be positive definite if 

x t Ax > 0 (A19) 


for all nonvanishing vectors x. It can be shown that a matrix is positive definite if the 
determinants of all its leading minors are positive. The leading minors of A are the 
n square matrices 


An A 12 • • • A 1 k 

A12 A22 • • • A 2 fc 


k = 1,2,n 


Ak 1 A k 2 • • • A kk 


Therefore, positive definiteness requires that 


An > 0, 


Au Au 
A 21 A 22 


An 

A\2 

A 13 




A 21 

A 22 

A 23 

> 0 ,.. 

-,\A\ > 0 

(A20) 

A 31 

A 32 

A 33 





Useful Theorems 

We list without proof a few theorems that are utilized in the main body of the text. 
Most proofs are easy and could be attempted as exercises in matrix algebra. 


(AB) r = B r A r 

(A21a) 

(AB)" 1 = B^A" 1 

(A21b) 

|A r | = |A| 

(A21c) 

|AB| = |A| |B| 

(A21d) 


if C = A r BA where B = B r , then C = C r 


(A21e) 
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EXAMPLE A4 

Letting 


'l 2 3~ 


r 


8 ' 

1 2 1 

U = 

6 

v = 

0 

0 1 2 


-2 


-3 


compute u + v, u • v, Av and u 7 Av. 


Solution 


1 + 8' 


9~ 

6 + 0 

= 

6 

1 

co 

1 

CN 

1 

_1 


-5 


u • v = 1(8) + 6(0) + (—2)(—3) = 14 



ai-v 


1 (8) + 2(0) + 3(—3)' 


’-l" 

Av = 

a 2 v 

= 

1(8) + 2(0)+ 1 (-3) 

= 

5 


a 3 v 


0 (8) + 1(0) + 2(—3) 


-6 


u r Av = u • (Av) = !(-!) + 6(5) + (-2)(-6) = 41 


EXAMPLE A5 

Compute ]A|, where A is given in Example A4. Is A positive delinite? 


Solution Laplace’s development of the determinant on the first rowyields 


2 1 

- 2 

1 1 

„ 1 2 
+ 3 

1 2 


0 2 

0 1 


= 1(3)-2(2)+3(1) =2 


Development on the third row is somewhat easier due to the presence of the zero 
element: 


2 3 

- 1 

1 3 

+ 2 1 

2 

2 1 


1 1 

1 

2 


= 0 (— 4 ) - 1 (— 2 ) + 2 ( 0 ) = 2 


To verify positive definiteness, we evaluate the determinants of the leading 
minors: 


An = 1 > 0 O.K. 



Al2 


1 2 

^21 

A-22 


1 2 


Not O.K. 


A is not positive definite. 
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EXAMPLE A6 

Evaluate the matrix product AB, where A is given in Example A4 and 


B = 


-4 

1 

2 


1 

-4 

-2 


Solution 


AB = 


ai bj 

a 2 bj 
a 3 bj 


ai b 2 
a 2 b 2 
a 3 b 2 


1(—4) +2(1) +3(2) 
1 (—4) + 2(1) + 1(2) 
0(—4) + 1(1) + 2(2) 


1(1) + 2 (—4) + 3(—2) 
1(1) + 2 (—4) + 1 (—2) 
0 ( 1 ) + 1 (— 4 ) + 2 (— 2 ) 


4 -13 

0 -9 

5 -8 
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operations on, 21-22 
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backward linite difference approximations, 
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banded matrix, 56-66 
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bisection method, for equation root, 145-148 
brent, 150-151 
Brent’s method, 148-153 
Bulirsch-Stoer algorithm, 286-290 
Bulirsch-Stoer method, 278-279, 283 
algorithm, 286-290 
midpoint method, 283-284 
Richardson extrapolation, 284-286 
bulStoer, 287 
byte code, 1 

choleski(a), 47-48 
Choleski’s decomposition, 45-52 
cmath module, 18-19 

coefficient matrices, symmetric/banded, 56-66 
symmetric, 59-60 
symmetric/pentadiagonal, 61-66 
tridiagonal, 57-59 
comparison operators, in Python, 8 


composite Simpson’s 1/3 rule, 204 
composite trapezoidal rule, 200-201 
conditionals, in Python, 9 
conjGrad, 86, 90 

conjugate gradient methods, 86, 88-96, 389-400 
conjugate directions, 390-391 
Fletcher-Reeves method, 396-400 
PowelTs method, 391-396 
continuation character, 6 
cubicSpline, 117-119 
cubic splines, 115-121, 195 
curve fitting. See interpolation/curve fitting 
cyclic tridiagonal equation, 94 

deflation of polynomials, 172-173 
diagonal dominance, 68 
Doolittle’s decomposition, 42-45 
dot, 3 

eigenvals3, 371—372 
eigenvalue problems. See symmetric matrix 
eigenvalue problems 
elementary operations, linear algebra, 31 
embedded integration formula, 275 
equivalent linear equation, 31 
error 

control in Python, 15 

in finite difference approximations, 186-187 
Euhler’s method, stability of, 272 
Euler-Maclaurin summation formula, 201 
evalPoly, 172 

evaluation of polynomials, 170-172 
exponential functions, fitting, 130-131 

false position method, roots of equations, 179 
finite difference approximations, 181-187 
errors in, 186-187 
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finite difference approximations ( cont .) 
flrst Central difference approximations, 
182-183 

flrst noncentral, 183-184 
second noncentral, 185 
finite elements, 233 
flrst Central difference approximations, 

182-183 

flrst noncentral finite difference 
approximations, 183-184 
fletcherReeves, 397-398 
Fletcher-Reeves method, 396-400 
forward finite difference approximations, 183 
fourth-order differential equation, 316-320 
fourth-order Runge-Kutta method, 257-265 
functions, in Python, 15-16 

gaussElimin, 37—38 
Gauss elimination method, 33-41 
algorithm for, 35-38 
back substitution phase, 37 
elimination phase, 35-36 
back substitution phase, 35 
elimination phase, 34-35 
multiple sets of equations, 38-41 
Gauss elimination with scaled row pivoting, 
68-76 

Gaussian integration, 215-230 
abscissas/weights for Guaussian quadratures, 
221-230 

Gauss-Chebyshev quadrature, 222 
Gauss-Hermite quadrature, 223 
Gauss-Laguerre quadrature, 222-223 
Gauss-Legendre quadrature, 221-222 
Gauss quadrature with logarithmic 
singularity, 224 

determination of nodal abscissas/weights, 
219-221 

formulas for, 215 
orthogonal polynomials, 217-219 
Gauss-Jordan elimination, 32 
Gauss-Legendre quadrature over quadrilateral 
element, 233-240 
gaussNodes, 224-225 
gaussPlvot,71—72 
gaussQuad, 225—226 
gaussQuad/?, 235—237 
gaussSeidel, 87-88 
Gauss-Seidel method, 85-88 
gerschgorin, 368—369 
Gerschgorin’s theorem, 368-369 
golden section search, 383-389 
goldSearch, 385—386 


Higher-order equations, shooting method, 
301-307 

householder, 362—363 

householder reduction to tridiagonal form, 
357-365 

accumulated transformation matrix, 
361-365 

householder matrix, 358-359 
householder reduction of symmetric 
matrix, 359-361 

Idle (code editor), 3 
ill-conditioning, 29-30 

incremental search method, roots of equations, 
143-145 

indirect methods. See iterative methods 
initial value problems 
adaptive Runge-Kutta method, 275-283 
Bulirsch-Stoer method, 278-279, 283 
algorithm, 286-290 
midpoint method, 283-284 
Richardson extrapolation, 284-286 
introduction, 248-249 
multistep methods, 294 
problem set, 266-271,290-294 
Runge-Kutta methods, 255-265 
fourth-order, 257-265 
second-order, 255-257 
stability/stiffness, 271-275 
stability of Euhler’s method, 272 
stiffness, 272-275 
Taylor series method, 249-255 
Input/output 
printing, 14-15 
reading, 13 
integration order, 234 
interpolation, derivatives by, 190-195 
cubic spline interpolant, 195 
polynomial interpolant, 190-191 
interpolation/curve fitting 
interpolation with cubic spline, 115-121 
introduction, 103 
least-squares fit, 124-137 
fitting a straight line, 126 
fitting linear forms, 126-127 
polynomial fit, 127-129 
weighting of data, 129-137 
fitting exponential functions, 

130-131 

weighted linear regression, 130 
polynomial interpolation, 104-115 
Lagrange’s method, 104-105 
limits of, 110-115 
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Neville’s method, 108-110 
Newton’s method, 105-108 
problem set, 121-124,137-140 
rational function interpolation, 140-141 
interval halving method. See bisection method 
inversePower, 345-346 
inversePower3, 373-374 
iterative methods, 85-96 
conjugate gradient method, 86, 88-96 
Gauss-Seidel method, 85-88 

jacobi, 331 
Jacobian matrix, 235 
Jacobi method, 326-342 
Jacobi diagonalization, 328-333 
Jacobi rotation, 327-328 
similarity transformation/diagonalization, 
326-327 

transformation to Standard form, 

333-342 

Jenkins-Traub algorithm, 179-180 

knots of spline, 115 
Kronecker delta, 104 

Lagrange’s method, 104-105 
Laguerre’s method, 173-178 
laraRange, 369-370 
LAPACK (Linear Algebra PACKage), 27 
least-squares fit, 124-137 
fitting linear forms, 126-127 
fitting straight line, 126 
polynomial fit, 127-129 
weighting data, 129-137 
fitting exponential functions, 

130-131 

weighted linear regression, 130 
linear algebraic equations Systems. See also 
matrix algebra 
back substitution, 32 
direct methods overview, 31-33 
elementary operations, 31 
equivalent equations, 31 
forward substitution, 32 
Gauss elimination method, 33-41 
algorithm for, 35-38 
back substitution phase, 37 
elimination phase, 35-36 
back substitution phase, 35 
elimination phase, 34-35 
multiple sets of equations, 38-41 
ill-conditioning, 29-30 
introduction, 27-33 


iterative methods, 85-96 
conjugate gradient method, 86, 88-96 
Gauss-Seidel method, 85-88 
linear Systems, 30 
LU decomposition methods, 41-52 
Choleski's decomposition, 45-52 
Doolittle’s decomposition, 42-45 
matrix inversion, 82-84 
methods of solution, 30-31 
notation in, 27-28 
pivoting, 66-76 
diagonal dominance, 68 
Gauss elimination with scaled row pivoting, 
68-76 

when to pivot, 74-76 
problem set, 53-56, 77, 86, 96-101 
QR decomposition, 101 
singular value decomposition, 101 
symmetric/banded coefficient matrices, 
56-66 

symmetric coefficient, 59-60 
symmetric/pentadiagonal coefficient, 

61-66 

tridiagonal coefficient, 57-59 
uniqueness of solution, 28-29 
linear forms, fitting, 126-127 
linear Systems, 30 
linlnterp, 297 
lists, 5-7 
loops, 10-11 
LR algorithm, 380 
LUdecomp, 44 
LUdecomp3, 59 
LUdecomp5, 66 

LU decomposition methods, 41-52 
Choleski’s decomposition, 45-52 
Doolittle’s decomposition, 42-45 
LUpivot, 72—73 

mathematical functions, 12-13 
math module, 17-18 
MATLAB, 2-3 
matrix algebra, 412-417 
addition, 413 
determinant, 414-415 
example, 416-417 
inverse, 414 
multiplication, 413-414 
positive definiteness, 415 
transpose, 413 
useful theorems, 415 
matrix inversion, 82-84 
methods of feasible directions, 408 
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methods of solution, 30-31 
midpoint,285—286 
minimization along line, 383-389 
bracketing, 383 

golden section search, 383-389 
modules, in Python, 17 
multiple integrals, 233 
Gauss-Legendre quadrature over 

quadrilateral element, 233-240 
quadrature over triangular element, 
240-245 

multistep methods, for initial value problems, 
294 

NameError, 23 
Namespace, 23 
natural cubic spline, 115 
Nelder-Mead method, 407 
neville,109-110 
Neville’s method, 108-110 
Newton-Cotes formulas, 199-207 
composite trapezoidal rule, 200-201 
recursive trapezoidal rule, 202-203 
Simpson’s rules, 203-207 
trapezoidal rule, 200 
newtonPoly, 107—108 
newtonRaphson, 155—156 
newtonRaphson2, 160—161 
Newton-Raphson method, 153-158 
norm of matrix, 29 
notation, 27-28 
Numarray module, 3 
numarry module, 19—23, 7 
accessing/changing array, 21 
array functions, 22-23 
copying arrays, 23 
creating an array, 19-21 
operations on arrays, 21-22 
numerical differentiation 
derivatives by interpolation, 190-195 
cubic spline interpolant, 195 
polynomial interpolant, 190-191 
finite difference approximations, 

181- 187 

errors in, 186-187 

first Central difference approximations, 

182- 183 

first noncentral, 183-184 
second noncentral, 185 
introduction, 181 
problem set, 195-197 
Richardson extrapolation, 187-190 
numerical instability, 243,262 


numerical integration 
Gaussian integration, 215-230 
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quadratures, 221-230 
Gauss-Chebyshev quadrature, 222 
Gauss-Hermite quadrature, 223 
Gauss-Laguerre quadrature, 222-223 
Gauss-Legendre quadrature, 221-222 
Gauss quadrature with logarithmic 
singularity, 224 

determination of nodal abscissas/weights, 
219-221 

formulas for, 215 
orthogonal polynomials, 217-219 
introduction, 198-199, 217 
multiple integrals, 233 
Gauss-Legendre quadrature over 
quadrilateral element, 233-240 
quadrature over triangular element, 
240-245 

Newton-Cotes formulas, 199-207 
composite trapezoidal rule, 200-201 
recursive trapezoidal rule, 202-203 
Simpson’s rules, 203-207 
trapezoidal rule, 200 
problem set, 212-215, 230-233, 245-247 
Romberg integration, 207-212 

operators 
arithmetic, 7 
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optimization 

conjugate gradient methods, 389-400 
conjugate directions, 390, 391 
Fletcher-Reeves method, 396-400 
Powell's method, 391-396 
introduction, 381-383 
minimization along line, 383-389 
bracketing, 383 

golden section search, 383-389 
Nelder-Mead method, 407 
problem set, 401-407 
simplex method, 407-408 
simulated annealing method, 407 
orthogonal polynomials, 217-219 
overrelaxation factor, 86, 96-101 

piecewise cubic curve, 116 
pivoting, 66-76 
diagonal dominance, 68 
Gauss elimination with scaled row pivoting, 
68-76 

when to pivot, 74-76 
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polyFit,128—129 
polynomial fit, 127-129 
polynomial interpolant, 190-191 
polynomial interpolation, 104-115 
Lagrange’s method, 104-105 
limits of, 110-115 
Neville’s method, 108-110 
Newton’s method, 105-108 
polynomials, zeroes of, 170-178 
deflation of polynomials, 172-173 
evaluation of polynomials, 170-172 
Laguerre’s method, 173-178 
polyRoots, 174-176 
powell, 393-394 
Powell's method, 391-396 
Prandtl stress function, 235 
printing input, 14-15 
printSoln, 251—252 
Python 

arithmetic operators, 7 
craath module, 18-19 
comparison operators, 8 
conditionals, 9 
error control, 15 
functions, 15-16 
general information, 1-4 
obtaining Python, 3-4 
overview, 1-3 
lists, 5-7 
loops, 10-11 

mathematical functions, 12-13 
math module, 17-18 
modules, 17 
numarry module, 19—23 

accessing/changing array, 21 
array functions, 22-23 
copying arrays, 23 
creating an array, 19-21 
operations on arrays, 21-22 
printing input, 14-15 
reading input, 13 
scoping of variables, 23-24 
strings, 5 
tuples, 5 

type conversion, 11-12 
variables, 4 

writing/running programs, 24-26 
Python interpreter, 1, 1 

QR algorithm, 380 

quadrature. See numerical integration 
quadrature over triangular element, 
240-245 


rational function interpolation, 140-141 
reading input, 13 

recursive trapezoidal rule, 202-203 
relaxation, 86 
relaxation factor, 86, 88-96 
Richardson extrapolation, 187-190, 284-286 
romberg, 209-210 
Romberg integration, 207-212 
rootsearch, 144 
roots of equations 
Brent’s method, 148-153 
false position method, 179 
incremental search method, 143-145 
introduction, 142-143 
Jenkins-Traub algorithm, 179-180 
method of bisection, 145-148 
Newton-Raphson method, 153-158 
problem set, 164-169,178-179 
secant method, 179 
Systems of equations, 158-163 
Newton-Raphson method, 159-163 
zeroes of polynomials, 170-178 
deflation of polynomials, 172-173 
evaluation of polynomials, 170-172 
Laguerre's method, 173-178 
row pivoting, 56, 67 

Runge-Kutta-Fehlberg formulas, 275-276 
Runge-Kutta methods, 255-265 
fourth-order, 257-265 
second-order, 255-257 
run_kut4,258—259 
run_kut5, 278—279, 283 

scaled row pivoting, 68-76 
secent formula, 179 
second forward finite difference 
approximations, 185 
second noncentral finite difference 
approximations, 185 

second-order differential equation, 296-301, 
311-316 

second-order Runge-Kutta method, 255-257 
shape functions, 234 
shooting method, 296-307 
higher-order equations, 301-307 
second-order differential equation, 296-301 
Shur’s factorization, 380 
similarity transformation, 104 
Simpson’s 3/8 rule, 204 
Simpson’s rules, 203-207 
slicing operator, 3 
sortjacobi, 333 
sparsely populated matrix, 56 
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stabiiity/stiffness, 271-275 
stability of Euhler's method, 272 
stiffness, 272-275 
stdForm, 335—336 
stiffness, 272-275 
straight line, fitting, 126 
strings, 5 

Strum sequence, 365-375 
sturmSeq, 366-367, 375 
swapCols, 70 
swapRows, 70 

symmetric/banded coefficient matrices, 56-66 
symmetric coefficient, 59-60 
symmetric/pentadiagonal coefficient, 61-66 
tridiagonal coefficient, 57-59 
symmetric coefficient matrix, 59-60 
symmetric matrix eigenvalue problems 
eigenvalues of symmetric tridiagonal 
matrices, 365-375 
bracketing eigenvalues, 369-371 
computation of eigenvalues, 371-372 
computation of eigenvectors, 373-375 
Gerschgorin’s theorem, 368-369 
Strum sequence, 365-375 
householder reduction to tridiagonal form, 
357-365 

accumulated transformation matrix, 
361-365 

householder matrix, 358-359 
householder reduction of symmetric 
matrix, 359-361 
introduction, 324-326 
inverse power/power methods, 342-351 
eigenvalue shifting, 344 
inverse power method, 342-344 
power method, 345-351 
Jacobi method, 326-342 
Jacobi diagonalization, 328-333 
Jacobi rotation, 327-328 
similarity transformation/diagonalization, 
326-327 

transformation to Standard form, 333-342 
LR algorithm, 380 
problem set, 351-357, 375-380 
QR algorithm, 380 
Shur's factorization, 380 


symmetric/pentadiagonal coefficient matrix, 
61-66 

synthetic division, 172-173 
Systems of equations 
Newton-Raphson method, 159-163 
roots of equations, 158-163 

taylor, 250-251 
Taylor series, 249-255, 409-412 
function of several variables, 410-412 
function of single variable, 409-410 
transpose operator, 413 
trapezoid, 202—203 
trapezoidal rule, 200 
triangleQuad, 242 
tridiagonal coefficient matrix, 57-59 
tuples, 5 

two-point boundary value problems 
finite difference method, 310-320 
fourth-order differential equation, 316-320 
second-order differential equation, 

311-316 

introduction, 295-296 
problem set, 307-310, 320-323 
shooting method, 296-307 
higher-order equations, 301-307 
second-order differential equation, 296-301 
type (a), 13 
type conversion, 11-12 

underrelaxation factor, 86, 90 

variables 
Python, 4 
scoping, 23-24 

Taylor series, 409-410, 410-412 

weighted linear regression, 130 
writing/running programs, in Python, 24-26 

ZeroDivisionError, 15 
zeroes of polynomials, 170-178 
deflation of polynomials, 172-173 
evaluation of polynomials, 170-172 
Laguerre’s method, 173-178 
zero offset, 3 




